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The RICIS Concept 


The University of Houston-Clear Lake established the Research Institute for 
Computing and Information Systems (RICIS) in 1 986 to encourage the NASA 
Johnson Space Center (JSC) and local industry to actively support research 
in the computing and information sciences. As part of this endeavor, UHCL 
proposed a partnership with JSC to jointly define and manage an integrated 
program of research in advanced data processing technology needed for JSC's 
main missions, including administrative, engineering and science responsi- 
bilities. JSC agreed and entered into a continuing cooperative agreement 
with UHCL beginning in May 1986, to jointly plan and execute such research 
through RICIS, Additionally, under Cooperative Agreement NCC 9-16, 
computing and educational facilities are shared by the two institutions to 
conduct the research. 

The UHCL/RICIS mission is to conduct, coordinate, and disseminate research 
and professional level education in computing and information systems to 
serve the needs of the government, industry, community and academia. 
RICIS combines resources of UHCL and its gateway affiliates to research and 
develop materials, prototypes and publications on topics of mutual interest 
to its sponsors and researchers. Within UHCL, the mission is being 
implemented through interdisciplinaiy involvement of faculty and students 
from each of the four schools: Business and Public Administration, Educa- 
tion, Human Sciences and Humanities, and Natural and Applied Sciences. 
RICIS also collaborates with industry in a companion program. This program 
is focused on serving the research and advanced development needs of 
industry. 

Moreover, UHCL established relationships with other universities and re- 
search organizations, having common research interests, to provide addi- 
tional sources of expertise to conduct needed research. For example, UHCL 
has entered into a special partnership with Texas A&M University to help 
oversee RICIS research anl education programs, while other research 
organizations are involved via the “gateway* concept 

A major role of RICIS then is to find the best match of sponsors, researchers 
and research objectives to advance knowledge in the computing and informa- 
tion sciences. RICIS, working jointly with its sponsors, advises on research 
needs, recommends principals for conducting the research, provides tech- 
nical and administrative support to coordinate the research and integrates 
technical results into the goals of UHCL, NASA/ JSC and industry. 
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EXECUTIVE SUMMARY 


Objectives of the Study 

The Shuttle Data Systems Branch (SDSB) of the Flight Data Systems Division (FDSD) at Johnson Space 
Center contracted with Southwest Research Institute (SwRI) to validate the effectiveness of an interactive 
video course on the code inspection process. The purpose of this project was to determine if this course 
could be effective for teaching NASA analysts the process of code inspection. In addition, NASA was 
interested in the effectiveness of this unique type of instruction (Digital Video Interactive^), for providing 
training on software processes. 

Conclusions 

This study found the Carnegie Mellon course, "A Cure for the Common Code", effective for teaching 
the process of code inspection. In addition, analysts prefer learning with this method of instruction, or 
this method in combination with other methods. As is, the course is definitely better than no course at 
all; however, findings indicate changes are needed. Following are conclusions of this study: 

• The course is instructionally effective. 

• The simulation has a positive effect on student’s confidence in their ability to apply new 
knowledge. 

• Analysts like the course and prefer this method of training, or this method in combination with 
current methods of training in code inspection, over the way training is currently being 
conducted. 

• Analysts responded favorably to information presented through scenarios incorporating full 
motion video. 

• Some course content needs to be changed. 

• Some content needs to be added to the course. 

Recommendations 

SwRI believes this study indicates interactive video instruction combined with simulation is effective for 
teaching software processes. Based on the conclusions of this study, SwRI has outlined seven options 
for NASA to consider. SwRI recommends the option which involves creation of new source code and 
data files, but uses much of the existing content and design from the current course. Although this option 
involves a significant software development effort, SwRI believes this option will produce the most 
effective results. 
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1.0 INTRODUCTION 


1.1 Purpose 

Analysts in the NASA Flight Data Systems Division (FDSD) at Johnson Space Center manage the 
software configuration for the maintenance of shuttle software. NASA management recognizes the need 
for effective, efficient training to provide these analysts with the knowledge and skills necessary to 
perform their jobs. 

The purpose of this study was to determine the effectiveness of an interactive video course on the code 
inspection process. "A Cure for the Common Code", developed by the Software Engineering Institute 
at Carnegie Mellon University, consists of training modules, a reference library and simulated code 
inspections. In addition to determining the effectiveness of this particular piece of courseware, 
management is also interested in the effectiveness of this unique type of instruction (DVI®) for teaching 
content relevant to NASA needs. To investigate these issues, the NASA Shuttle Data Systems Branch 
(SDSB) contracted with the Training Systems and Simulators Department at Southwest Research Institute 
(SwRI) to conduct this study. 


1.2 Overview of the Study 

1.2.1 Methodology 

SwRI created a plan to validate the effectiveness of the Carnegie Mellon code inspection course. The 
plan consisted of two parts: validation of the course content and validation of the instructional 

effectiveness of the course. Validation of the content was achieved by comparing code inspection 
objectives and comparing code inspection models. Validation of the effectiveness of the course was 
measured by testing knowledge of information, application of information, and by gathering analysts’ 
reactions to the course. 

Three analysts participated in the study, one from NASA and two from IBM. The analysts’ backgrounds 
were varied in terms of computer experience, computer use, programming experience, experience with 
code inspection, and job responsibilities. Analysts with diverse backgrounds were sought in order to 
gather different perspectives on the code inspection course. 

The materials used for this validation study included the inspection course, "A Cure for the Common 
Code". To measure the effectiveness of the course, SwRI developed instruments including: 
pretest/posttest, analyst questionnaire, interview questions, observation form, and demographic data sheet. 

1.2.2 Findings 

Findings for the study are presented as content review findings and instructional effectiveness review 
findings. Content review findings state results from the review of course objectives, course content and 
underlying process models for code inspection. Instructional effectiveness review findings report results 
in terms of gain scores between the pretest and Posttest 1 (knowledge of information), gain scores 
between Posttest 1 and Posttest 2 (application of information), program feedback, analyst responses to 
the questionnaire and interview, and evaluator observations. Finally, a summary of findings is presented 
in terms of strengths and weaknesses of the course. 



1.2.3 Conclusions 


Conclusions based on the findings are presented. Conclusions are presented for course content, 
instructional effectiveness of the course, and analyst opinions about the course. 

1.2.4 Limitations 

There were known limitations for this study which may have affected results. Limitations of the study 
are presented, as well as limitations for the course ("A Cure for the Common Code"). Limitations of 
the course as cited in documentation by Carnegie Mellon are also included. 

1.2.3 Recommendations 

Based on the conclusions of this study, SwRI has outlined seven options for NASA to consider. These 
options are presented along with pros and cons for each. SwRI recommends one option believed to be 
the most instructionally effective and most cost effective method for incorporating process simulation 
training into current training efforts. 
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2.0 METHODOLOGY 


2.1 Approach of the Study 

The plan created by SwRI to validate the effectiveness of Carnegie Mellon’s code inspection course 
consisted of two parts. The objective of the first part was to validate the content of the course. The 
objective of the second part was to validate the instructional effectiveness of the course (including 
presentation strategy). Below is a summary of the validation plan. Appendix G contains a copy of the 
plan. 

2.1.1 Validate Content of the Course 

Content validity of the code inspection course was measured in two ways. First, instructional objectives 
were compared and second, models for code inspection were compared. 

2. 1 . 1 . 1 Compare Code Inspection Objectives 

NASA objectives for the code inspection process, as indicated by descriptions in the Software Formal 
Inspections Guidebook (August, 1991) and the NASA Software Inspection Process Standard (December 
9, 1991), were compared with Carnegie Mellon course objectives. 

SwRI analysts took high-level objectives provided by Carnegie Mellon (see Appendix D) and added a fine 
level of detail from the course. SwRI took these annotated course objectives (see Appendix C) and 
compared them with the description of code inspection provided in the NASA documents. These same 
annotated objectives were given to IBM who, based on their experience, also compared them with the 
NASA code inspection process. 

2.1. 1.2 Compare Models for Code Inspection 

Second, models for code inspection were compared. The model for code inspection used by NASA 
(Software Formal Inspections Guidebook and NASA Software Inspection Process Standard) was compared 
with the model used by Carnegie Mellon in the code inspection course (see Appendix E). Again, this 
comparison was done by both SwRI and verified by IBM. 

2.1.2 Validate Effectiveness of the Course 

Validation of instructional effectiveness involved three analysts using the code inspection course. Each 
analyst spent approximately six hours during one day working through the course (including tests, 
questionnaire and interview). SwRI chose three indicators of effectiveness for the code inspection course: 

* knowledge of information 

* application of information 

* reactions to the course 
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2. 1 .2. 1 Knowledge of Information 

Analysts were given a pretest prior to receiving instruction. Upon completion of the course instructional 
modules, analysts were given a posttest. The purpose of this posttest was to measure knowledge of 
information as a result of having completed the instructional modules. 

2. 1.2.2 Application of Information 

The course also simulates a code inspection. Analysts were given the opportunity to assume a role 
(recorder) and participate in a simulated code inspection. They were called upon to apply the knowledge 
and skills learned from the instructional modules in the training room. Course feedback was given to 
indicate analysts’ ability to apply the information learned. In addition, a second posttest, given after 
completion of the simulation, measured any net gain or loss of information that may have resulted from 
the simulated experience. 

2.1. 2. 3 Analyst Opinions 

After completion of the entire code inspection course, analysts were administered a questio nnair e and 
were then interviewed. The purpose of these two activities was to deter mine the analysts’ reactions to 
the course, including their likes, dislikes, and suggestions for improvement and use. 


2.2 Subjects 

Three analysts participated in the study, one from NASA (FDSD) and two from IBM. Following is a 
summary of die analysts’ backgrounds. A complete description of demographic information on the three 
analysts is provided in Appendix B. 

Each analyst had between five and fourteen years of computer experience with between one and six hours 
of use per day. Computer usage included word processing, progr amming , and other applications. F adi 
analyst had taken from one to seven college level software courses. None of the analysts had received 
any formal college instruction in die Ada programming language; however, other languages included 
Pascal, C, Cobol, LSP, Basic, Fortran, and informal instruction in Ada. Only one analyst had 
participated in a code inspection before, but two analysts indicated their job may require it in the future. 
None of the three analysts had ever received formal training in die code inspection process. 

All three analysts had some experience with training via computer in the past, ranging from one to eight 
courses. They all felt it was an effective method for learning and had a positive attitude toward it. 
Interaction was cited as a major advantage of computer based training, as well as the feeling that it was 
much more interesting than reading from a manual. 
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23 Materials 


2.3.1 Carnegie Mellon Course 

The Carnegie Mellon course, "A Cure for the Common Code”, is based on a fictitious company named 
"Ultimex". The course teaches the process of code inspection as if the student is a new employee. The 
course uses full motion video, still video, audio, and simulation. 

Various rooms in the company are available to the user. The training room consists of five instructional 
modules: 


• Formal Inspections: Purpose and Process 

• Inspection Types and Differences 

• Inspection Roles and Pitfalls 

• Inspection Tools and Forms 

• Inspection Communications 

The conference room is where simulated code inspections occur. The code inspection simulation allows 
the student to apply what he/she learned from the instructional modules. The simulation uses a rule-based 
expert system of approximately one hundred rules. The expert system determines the responses of the 
simulated personalities, controls dialogue, interprets user responses, and controls the visual presentation. 

Other rooms in the company are also available to the user. An overview of die course is given in the 
auditorium. A library is available for reference, which includes articles and manuals, as well as 
videotapes. The user has an office with tools available for reviewing code in preparation for a code 
inspection. Finally, to make the environment more realistic, the secretary’s office and coffee room are 
also included. An outline of the course can be found in Appendix H. 

2.3.2 Instruments 

In addition to the course, a number of other materials were used in this study. Five instruments were 
created by SwRI to measure the effectiveness of the course, as well as reactions of the analysts 
participating in the study. The following instruments were created: pretest/posttest, analyst 

questionnaire, interview questions, an observation form, and a demographic data sheet. Brief descriptions 
of these instruments follow. Samples of these instruments appear in Appendix A. 

2.3.2. 1 Pretest/Posttest 

The pretest and posttest (Posttest 1 and Posttest 2) were the same instrument. They were created by an 
instructional designer with pretest trials using three subjects that represented the target audience (software 
engineer, mathematician, engineer). The pretest trial results were reviewed by a subject matter expert 
(software engineer). 

The purpose of the pretest/posttest was determined by when it was administered in the study. The same 
instrument was used three times, since time and resource constraints did not permit creation of different 
versions of the test. 
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When administered prior to the study (pretest), the instrument served to give a baseline measure of 
knowledge of information, so any gain in score after instruction could be measured. When used as a 
posttest for the first tune (Posttest 1 , after the instructional modules), the instrument served as a measure 
of knowledge of information. When used as a posttest for die second time (Posttest 2, after the simulated 
code inspection), the instrument measured gain in knowledge as a result of application. The purpose was 
to determine if analysts learned any more from the simulation. 

The pretest/posttest instrument had three parts including definitions, fill in the blank, and multiple choice. 
Part 1, consisting of five questions, required analysts to define terms related to the code inspection 
process. Part 2, consisting of twenty fill in the blank questions, required analysts to recall information 
presented in the course. Part 3, consisting of forty multiple choice questions, required analysts to choose 
the correct answer(s) from the choices provided. 

2.3.2. 2 Analyst Questionnaire 

The analyst questionnaire was administered after completion of the course. The purpose of the 
questionnaire was to measure analysts’ reactions to the course and recommendations for change and use. 

The analyst questionnaire consisted of a total of thirty-eight statements. These statements were grouped 
into five categories including overall evaluation of the course (nine statements), course content (five 
statemen t s), learning effectiveness (two statements), instructional presentation (twelve statements), and 
system capabilities (ten statements). Analysts were asked to rate each statement on a scale of one to five 
(one indicating strong disagreement and five indicating strong a gnwngnt with the statement). 

2.3.2.3 Interview Questions 

The interview was conducted after completion of the course and immediately after the anal yst 
questionnaire. The purpose of the interview was to further expand on information collected by the 
questionnaire regarding analysts’ reactions to the course and recommendations for change anH use. 

Interview questions were grouped using the same five categories as the analyst questionnaire: overall 
evaluation of the course (ten questions), course content (seven questions), learning effectiveness (three 
questions), instructional presentation (four questions), and system capabilities (two questions). A total 
of twenty-six questions were included in the interview. 

2.3.2.4 Observation Form 

The observation form was used during data collection when analysts were actually using the course. The 
purpose of the observation form was to note any comments or actions during the use of the course which 
might be used to explain findings. During the entire use of the course, SwRI observers noted 
observations in four areas: problems or difficulties experienced, analyst comments, observer comments, 
time spent working on each task (modules, tests, simulation, etc.). 
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2.3.2.S Demographic Data Sheet 


The demographic data sheet was administered by NASA prior to data collection. The purpose of the 
demographic data sheet was to aid in choosing analysts to participate in the study, as well as helping 
explain findings. 

The data sheet consisted of a total of sixteen questions grouped into four categories. The four categories 
of questions were computer experience (three questions), progr amming experience (three questions), code 
inspection process (seven questions), and general information (three questions). 


2.4 Procedures 

2.4. 1 Study Procedures 

At the beginning of this study, the code inspection course was obtained from Carnegie Mellon and loaded 
on an SwRI computer system for evaluation. Carnegie Mellon provided high-level instructional objectives 
and extensive detail was added by SwRI to create a set of annotated objectives. 

The validation plan (see Appendix G) was written and dictated how the study would proceed. The plan 
consisted of two phases. Phase one called for assessment of the content validity of the course. Phase 
two called for three analysts to use the course, so data could be collected on effectiveness of the course, 
as well as analyst reactions to the course. The steps used in data collection are outlined below. 

2.4.2 Data Collection Procedures 

Prior to conducting trial runs of the course, NASA administered the demographic data sheet and the 
pretest to seven analysts (3 NASA analysts, 4 IBM analysts). Based on die information collected, SwRI 
chose three analysts (1 NASA analyst, 2 IBM analysts) to participate in the study. The proposal dictated 
the number of individuals that would participate in the study (three). 

During the study, analysts were instructed how to proceed through the course. First, they were 
introduced to the course in the auditorium. Next, they participated in the five instructional modules in 
the training room. Upon completion of the modules, they were administered Posttest 1 to measure their 
knowledge of the information presented. After the posttest, analysts were allowed to explore the library, 
practice with the Ada code in their office, and finally participate in a simulated code inspection. Upon 
completion of the simulation, analysts were given a questionnaire requesting them to give their reactions 
as well as recommendations for the course. This questionnaire was followed by an interview with SwRI 
analysts. After completing the questionnaire and interview, the analysts were administered Posttest 2 (see 
Appendix A). 
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3.0 FINDINGS 


3.1 Results of Data Collection 

SwRI validated the relevance of this course to NASA’s training needs by examining the course, 
comparing the course with the existing NASA code inspection standards/practice and evaluating the 
course’s instructional effectiveness. This validation included an instructional design review of the subject 
matter (SwRI), subject ma tt er expert review (SwRI and IBM), and an instructional effectiveness review 
(SwRI), including target audience trials (NASA and IBM analysts). 

3.1.1 Content Review Findings 

The following paragraphs describe the findings from the review of objectives, course content and 
underlying process models for code inspection. 

3. 1 . 1 . 1 Content Review by SwRI 

SwRI compared the course content with the stated objectives from Carnegie Mellon. SwRI verified that 
the course content was consistent with die stated course objectives. 

SwRI compared code inspection objectives as outlined in two NASA documents (Software Formal 
Inspections Guidebook and NASA Software Inspection Process Standard) with die objectives of the 
course. In order to make comparisons consistent in scope and depth, SwRI extracted more detail from 
the course and annotated the Carnegie Mellon objectives. SwRI found the anrw->tat<*t course objectives 
to be very similar to the NASA code inspection guidelines and standards. 

3. 1 . 1 .2 Content Review by IBM 

At the request of SwRI and NASA, analysts from IBM reviewed the annotated course objectives. IBM 
experts evaluated the detailed outline with respect to established NASA code inspection practice and 
procedures. SwRI also demonstrated the course to IBM experts in order to enhance their analyses. The 
following summary lists discrepancies between the course and NASA’s code inspection process. The 
complete report from IBM appears in Appendix F. 

Areas where the course content conflicts: 

* characteristics of an inspection meeting 

* differences between inspection and walkthrough procedures 

* role of recorder 

Areas that require additional detail: 

* purpose of formal inspections 

* stages of the formal inspection process 

* benefits of the formal inspection process 

* role of planning and preparation in the inspection process (pl anning ) 

* role of planning and preparation in the inspection process (preparation) 
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• review roles assumed during inspection 

• types of checklists 

• basic rules for code inspections 

• moderator role description 

• checklists and forms use in the formal inspections process 

• importance of inspection as an organizational approach to ensure process 
stabil ity/improvement 

3. 1.1.3 Code Inspection Model Review by SwRI 

SwRI also compared models for code inspection. Carnegie Mellon provided a description of the model 
the course uses. The two documents previously mentioned, Software Formal Inspections Guidebook and 
NASA Software Inspection Process Standard, present NASA’s model. SwRI found the two models to 
be similar. Only minor differences were detected. 

3. 1.1.4 Code Inspection Model Review by IBM 

IBM reviewed the code inspection models and found the inspection process steps adequate. IBM moved 
the sequence of one step, exit criteria. IBM added the following required steps: 

• re-work 

• post meeting errors 

• collection of inspection meeting reports 

• submission of summary data to database 

• extraction of reports from database 

• summary metric data 

• FACI/CI summary data 

IBM added additional details to the following role descriptions: 

• manager 

• producer 

• moderator 

• recorder 

IBM added the following roles as other individuals involved in the NASA inspection process: 

• librarian 

• designer/tester 

• independent tester 

• consumer 
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3.1.2 Instructional Effectiveness Review Findings 


The following paragraphs describe the findings from the target audience trials (NASA and IBM) and the 
instructional effectiveness review (SwRI). This section states results as collected by various instruments; 
therefore, information may overlap by design (e.g., analyst Questionnaire and interview questions). Raw 
data can be found in Appendix B. 

3. 1 .2. 1 Pretest/Posttest 1 (Knowledge of Information) 

Analyst scores on foe pretest (administered prior to using foe course) were compared with their scores 
on Posttest 1 (administered after using foe instructional modules). The purpose of this comparison was 
to determine how much analysts learned from foe instructional modules (Knowledge of Information). 

There was a definite improvement in ability to define terms after completing foe instructional modules. 
When asked to answer questions on foe pretest, such as stating the purposes of formal in spect ions or 
listing foe stages of foe formal code inspection process, analysts had difficulty producing completely 
correct answers. For example, on foe pretest none of foe analysts were able to list all stages in foe formal 
code inspection process, whereas on Posttest 1 all analysts were able to list most, if not all, foe stages. 
Multiple choice scores for all three analysts improved substantially from foe pretest to Posttest 1 (see table 
below). Results for foe multiple choice part of foe test were consistent with results for foe definition and 
fill in foe blank parts of foe test. Gains in scores from foe Pretest to Posttest 1 indicate a s ub s tantial 
increase in knowledge of information about code inspection after completion of foe course instructional 
modules. 


Part 3 

(Multiple Choice) 

Analyst 1 

Analyst 2 

Analyst 3 1 

Pretest 

70% 

74% 

— 

78% 

Posttest 1 

88% 

86% 

83% 

Posttest 2 

89% 

87% 

87% 


Note: Scores indicate foe percent correct out of a total 1S6 questions. 


3. 1.2.2 Posttest 1/Posttest 2 (Application of Information) 

Analysts scores on Posttest 1 (administered after foe instructional modules) were compared with their 
scores on Posttest 2 (ad m i ni s t e r ed after foe simulation). The purpose of this comparison was to measure 
gain in knowledge as a result of application (did foe analysts learn additional information from foe 
simulation). 

Changes in response from Posttest 1 to Posttest 2 on foe definition part of foe test were not substantial. 
Little additional information, if any, was noted, nor was there any noticeable improvement in quality of 
response. Similar results held true for foe fill in foe blank part of foe test. Once again, very little 
additional information was noted on Posttest 2. No noticeable improvement in foe quality of response 
was de t ec te d. Scores on foe multiple choice part of foe test increased slightly from Posttest 1 to Posttest 
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2 (see table above). Results for the definition, fill in the blank, and multiple choice parts of the test were 
consistent. Gains in scores were small, which indicates the analysts probably did not gain significant 
knowledge of information as a result of participating in the simulation. 

3. 1.2.3 Program Feedback 

The course provides some feedback upon completion of a simulated code inspection. Strengths and 
weaknesses of an analyst’s performance are given. A summary of the three analysts feedback is provided 
below. A complete listing of feedback for each analyst can be found in Appendix B. 

A strength noted for two analysts was good use of emotional tone. All three analysts received praise for 
never introducing irrelevant topics, while two analysts were complimented for stopping tangents. Another 
analyst was praised for expressing a minority opinion and overall good participation. 

The one weakness cited for all three analysts was missing two of the biggest errors in code. In addition, 
two analysts changed their opinions incorrectly (possibly due to group pressure). A weakness of one 
analyst was lack of input or being too passive during the inspection, while another was cited as being too 
aggressive. Another analyst had difficulty with the talk interface and left too many topics open without 
a final resolution. 

3. 1 .2.4 Analyst Self Report 

Analysts’ reactions to the course were measured with an analyst questionnaire and an interview. Findings 
from these two instruments are presented below. 

3.1. 2.4.1 Analyst Questionnaire 

The analyst questionnaire was divided into five sections. Results are summarized by section. A complete 
listing of results is found in Appendix B. 

Overall Evaluation of the Course: 

Analysts had a very positive attitude toward the course. They liked this method of instruction and 
preferred it over the way information is currently being taught. Analysts thought the course had numerous 
strengths, as well as some weaknesses which could be overcome. Two analysts strongly agreed this code 
inspection course has potential for use at NASA. The third analyst, who disagreed, does not currently 
perform code inspections. All analysts agreed they would like to see more courses of this type offered 
by NASA. 

Course Content: 

Two analysts agreed the code inspection model used in the course was similar to what is used at NASA 
(one analyst did not respond). The content of the course was very important (relevant) to one analyst and 
not as much to the other analysts (this correlated with their level of involvement on a daily basis in the 
code inspection process). 

There were mixed feelings about the purpose of the course. Two analysts strongly agreed the purpose 
was clear, whereas one disagreed strongly. Two analysts agreed the content was academically 
challenging, whereas one analyst had no opinion. Finally, all three analysts agreed the level of detail in 
the course was appropriate for preparing someone to participate in a code inspection. 
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Learning Effectiveness: 

All three analysts agreed they learned from the code inspection course. All also agreed they learned more 
from this method of instruction on code inspection than from current methods of instruction. 

Instructional Presentation: 

All analysts agreed the course captured their attention. Two analysts agreed the specific objectives of 
the course were not clear. In addition, prerequisite skills were not clear to all users. One analyst 
strongly agreed the course material was clear and well organized; the other analysts disagreed. One 
analyst thought there was enough practice provided, one was neutral, and a third did not respond. One 
analyst agreed strongly that feedback was adequate, one bad no opinion and another did not respond. 
Two analysts agreed feedback was meaningful and one analyst did not respond. Analysts agreed the 
course’s assessment of their performance was fair and meaningful. 

All analysts strongly agreed that this method of instruction probably caused them to learn more than 
current methods for learning code inspection. All analysts strongly agreed they liked the method of 
instruction. Analysts also agreed the course was appropriate for their background and experience. 

System Capabilities: 

Two analysts agreed learning how to use the course was fairly easy, one strongly disagreed. Similarly, 
two analysts agreed learning how to use the simulation was fairly easy, one strongly disagreed. Similar 
reactions held true for remembering names and uses of commands. (Note: In the interview, analysts 
elaborated on how easy/difficult the course was to use and specified particular areas of difficulty.) 

One analyst was frustrated during parts of the course, one did not have an opinion, and one was not too 
frustrated. Two analysts felt the simulation was slow. All analysts liked having a great deal of control 
over the instruction. Analysts agreed strongly the graphics were interesting and effective. They agreed 
the quality of the motion video was good and that it added value to die course. 

3.1. 2.4.2 Interview 

The interview form was divided into the same five sections as the analyst questionnaire. The interview 
provided analysts an opportunity to elaborate on their responses on die questionnaire, as well as answer 
additional questions. Again, results are summarized by section. A complete listing of results is found 
in Appendix B. 

Overall Evaluation of the Course: 

Analysts liked this method of instruction. They all indicated they liked the full motion video sc enari os 
best. Other strengths included: method of instruction (you remember the content longer), the simulation, 
and the library. Analysts indicated they disliked the following items: too much text on the screen, audio 
interferes with the text in places, the user interface (only parts, e.g. the mouse), and the section on groups 
in Module 5. Other weaknesses of the course included: lots of material to cover in one day, no 
instructions on how to operate the course, lack of instruction on how to use the tools, and not enough 
review provided. Analysts felt the weaknesses could be overcome. All analysts believed the course has 
potential for use in teaching the code inspection process. 
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Analysts recommended modifications to make the course more effective and easier to use. Specific 
applications for using this course were as training for new hires, as a review for experienced analysts, 
or in a workstation available for reference at any time. In addition, analysts recommended that 
instruction similar to this course (method of instruction) be used by NASA to teach other content as well. 

Course Content: 

One analyst felt the content of this course was particularly relevant at the present time, whereas the other 
two had no immediate need. All considered roles, behavior guidelines, and interpersonal communication 
skills the most relevant parts of the course. All analysts considered the segment on family and social 
groups to be irrelevant. Additional information desired included a segment on active listening, more 
video examples, more detail on some topics, easy access to definitions for unfamiliar terms and 
summaries at the end of sections. If called upon to use the course as is, analysts would use parts of the 
course which relate to the specific application at hand and omit the part on family and social groups. If 
the course were used, analysts recommend using the course over a period of time, rather than all in one 
day. 

Learning Effectiveness: 

All analysts said they learned from the course. They indicated they learned about the code inspection 
process, roles of participants, and what it is like to attend a real code inspection (scenarios and 
simulation). Two analysts indicated they learned general information, not details, because of the volume 
of information contained in the course and the limited time for using the course in this study. 

Analysts agreed the content was appropriate for the course, but would main* some changes. Analysts did 
emphasize the need for the course to clearly state a purpose and objectives. 

Instructional Presentation: 

Two analysts stated they liked the motion video segments in the program best. In addition, other 
desirable aspects of the course were the simulation and the library. Analysts indicated the following items 
were least liked about the method of presentation: too many text screens, audio sometimes competes with 
the text, lengthy introductions the user was required to sit through. The simulation, tools, and natural 
language interface were indicated as the most difficult parts of the program to use. 

Analysts indicated they prefer this method of presentation for learning code inspection or this method in 
combination with current methods over the way code inspection is presently taught (OJT, manuals, 
working with experienced analysts). One analyst indicated the decision to use a course of this type would 
need to be weighed against cost. 

System Capabilities: 

Analysts indicated they had some difficulty with the natural language interface and how to use the tools. 
One analyst was particularly frustrated with the mouse and its placement on the screen. They did indicate 
that they felt these difficulties could be overcome. 

3. 1.2.5 Evaluator Observations 

SwRI observed NASA and IBM analysts using the course. These observations support the opinions and 
recommendations expressed in both the questionnaire and the interview. A complete listing of 
observations can be found in Appendix B. 


13 



3.2 Summary of Findings 


Following are the strengths and weaknesses of this course as summarized by SwRI after reviewing the 
course and using it with the NASA/IBM analysts. 

3.2. 1 Strengths of the Course 

Following is a summary of the strengths of this course. 

3.2. 1 . 1 Instructional Issues 

* Most of the content is easily understood. 

* The level of difficulty of the content is appropriate for die target audience. 

* Feedback on performance in the simulation is built into the program (simulation) 

* The presentation of the content through full motion video is motiv atio nal (scenarios). 

* Help is provided (although not context sensitive). 

* The user controls pace of the instruction in almost all cases. 

* The simulation helps the user apply what is learned. 

* A library is provided for reference. 

* The program provides a means for exiting at almost all times. 

3.2. 1.2 Aesthetic Issues 

* In most cases, types of screens are consistent to provide navigation for the user (menu 
screens, text screens, etc.). 

* Only one typographical error was found. 

3.2. 1.3 Technical Issues 

* The course execution is consistent. 

3.2.2 Weaknesses of the Course 

Following is a summary of die weaknesses of this course. 

3.2.2. 1 Instructional Issues 

* No purpose is stated for the course. 

* Objectives are not clearly stated for the overall program or individual sections. 

* Some content is missing and needs to be added. 

* Some content needs to be changed. 

* Some content see m s irrelevant to the course (e.g., family and social groups). 

* The model for code inspection used in the course is missing some steps. 

* The order of steps in the model for code inspection used in this course needs to be 

changed. 

* Directions for use of the course are not clearly stated. 

* Complete documentation about the course needs to be provided (e.g., what it is, how to 
use it). 
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• The instructional sequence is not clearly stated. 

• The instructional modules need to be more interactive. 

• The user does not always have control over when to move to the next screen (not 
consistent). 

• Better provision for reviewing sections needs to be included (some chunks of information 
could be smaller). 

• The course needs to provide summaries/reviews at the end of each instructional module. 

• The user needs to be able to pause and resume from that point. 

• The user needs a provision to exit at most times (especially during lengthy introductions). 

• No practice questions with feedback are given during the instructional modules. 

• Evaluation criteria for the simulation is not explained. 

• Help is not context sensitive. 

• The text competes with audio in places and is not used consistently. 

• The course needs better instruction on how to use the tools. 

• The natural language interface is difficult to use and little instruction is provided. 

3.2.2.2 Aesthetic Issues 

• Screens are often packed with too much text. 

• Color is not used effectively (text and background colors do not complement each other). 

• Sections of motion video (e.g., motion video in the simulation that is not full motion) are 
unnatural (better than still frame, but not full motion). 

• The mouse does not appear in a consistent location on die screen. 

3.2.2. 3 Technical Issues 

• There are a few bugs in running the program; however, for the most part these are stated 
as limitations of the program (it must also be recognized that this is a prototype 
program). 

• The program is slow in some places, especially the long simulation. 

• The quality of the video is not clear during the simulation (partial motion video is lower 
quality compared with full motion video capability). 

• The quality of the audio is poor is some places (e.g., simulation). 
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4.0 CONCLUSIONS 


4.1 Conclusions of the Study 

This study found the course, 'A Cure for the Common Code", effective for teaching the process of code 
inspection. In addition, analysts prefer learning with this method of instruction or this method in 
combination with current methods. The unmodified course is definitely considered better than no course 
at all; however, findings indicate changes are needed. Our conclusions regarding the content, 
instructional effectiveness, and analysts’ opinions about the course are presented below with a brief 
explanation for each. 

4.1.1 Content 

Conclusion: Some content needs to be added to the course. 

Explanation: SwRI concluded that the scope of the course is adequate; however, there needs 

to be more depth in some areas. IBM liked the course but stated some items 
definitely need to be added to die course to reflect NASA’s code inspection 
process (see Appendix F). 

Conclusion: Some course content needs to be changed. 

Explanation: In their review of the course objectives, IBM indicated that some information 

needed to be changed to customize the course to fit the NASA code inspection 
process (see Appendix F). 

Conclusion: The steps in the code inspection process need to be more complete to closely 

follow the NASA model. 

Explanation: In their review of the code inspection model, IBM indicated that some steps were 

missing from the Carnegie Mellon model and would need to be added to more 
accurately reflect the NASA model for code inspection (see Appendix F). 

Conclusion: The order of the steps in the Carnegie Mellon code inspection model need to be 

changed to more closely follow the NASA model. 

Explanation: In their review of the code inspection model, IBM indicated the order of steps 

did not accurately reflect the NASA model (see Appendix F). 

4. 1 .2 Instructional Effectiveness 

Conclusion: The course is instructionally effective. 

Explanation: Based on gain scores between the pretest and Posttest 1 , the course demonstrated 

an ability to teach the stated objectives. 

Conclusion: The simulation has a positive effect on students’ confidence in their ability to 

apply new knowledge. 

Explanation: Although there was no meaningful gain in scores between Posttest 1 and Posttest 

2 (no gain in knowledge of information as a result of application), anal yst 
comments strongly indicated they favor the opportunity to practice code 
inspection and receive feedback on their performance while still in the tr ainin g 
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environment. They strongly agreed with the concept but did not think this 
particular simulation was as effective as it could be. 

Conclusion: Program feedback indicates that subjects were able to apply their new knowledge. 

Explanation: All three analysts received positive feedback from the course on strengths they 

exhibited in the simulation. In addition, weaknesses were also presented. The 
weaknesses the course detected are consistent with limitations of the study and 
of the course tools. See Appendix B for specific program feedback. 

4. 1 .3 Analyst Opinions 

Conclusion: Analysts like the course and prefer this method of training, or this method in 

combination with current methods of training in code inspection, over the way 
training is currently being conducted. 

Explanation: Overall, analysts appreciated this method of training (incorporating simulation, 

full motion video, scenarios), although they bad reservations about many of the 
specifics of this particular course. These favorable reactions can be seen in both 
the analyst questionnaire and the interview responses. 

Conclusion: Analysts responded favorably to information presented through scenarios 

incorporating full motion video. 

Explanation: Analysts indicated throughout the course a desire to see more scenarios. This 

recommendation was emphasized again in the questionnaire and the interview. 
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5.0 LIMITATIONS 


This study had a number of limitations which may have affected results. These limitations are 
summarized below. 

5.1 Limitations of the Study 

• The small sample size precluded a formal statistical analysis of the data. 

• The pretest and the two posttests were exactly the same tests. To create different 
versions of tests that measure the same knowledge would have required considerably 
more resources. While retesting with the same test is a reliable way to measure a gain 
in knowledge, some of the gain on later scores may be attributed to a familiarity with the 
test. Also, a pretest can enhance learning by serving as an advance organizer for topics 
that the student should pay more attention to. 

• Each subject was required to complete the course trial in one day, instead of the way the 
course was originally designed. This may have negatively affected their performance due 
to fatigue. Ideally, students would only take a few lessons at a time and not sit through 
six hours of instruction, simulation, and testing in one day. 

• Due to time constraints during the course trials, minimal time was provided to examine 
the sample code prior to the simulation. This limitation could cause weaker performance 
in finding errors in the code during the simulation. 

• The subjects were not proficient in die Ada programming language. This limitation could 
cause weaker performance in finding errors in the code during the simulation. 

• For two subjects, code inspection was not part of their current job. This may have 
negatively affected their motivational interest in die course topic. 


5J Limitations of the Course 

* A "Cure for the Common Code* is a prototype course, not a polished product intended 
for distribution. The student instructions and supporting doc ument ation are very sparse. 

* The DVI hardware (7 board set) used to develop and deliver the course is outdated. 

* The options in the natural language interface are limiting. The user may not be able to 
construct the exact response desired from the options provided by the natural language 
interface. 

* The audio quality is lower than it could be with this technology. 

* Some of the simulation environment does not utilize full motion video. The sequenced 
still frame displays look unnatural and jerky. This method, however, is probably more 
effective than just displaying one still frame. 

* The interlaced monitor mode can result in eye fatigue. 


53 Limitations of the Course as dted by Carnegie Mellon 

5.3.1 General 


All inspection forms are not implemented in the program. 

There are intermittent problems with some CD-ROM drives ("Critical Error Occurred"). 


18 



5.3.2 Training Room 

• Early exits are not available from the orientation session within the training room and 
from the tool descriptions within Module 4. 

• The student cannot exit the practice inspection and return later to the same state. This 
feature is available for the actual simulation. 

5.3.3 Library 

• The text materials in the library are incomplete. 

• The user is forced to sit through the orientation session in the library during the first 
visit. 

5.3.4 Conference Room 

• The instructions for using the talk interface are minimal. 

• The quality of the audio in parts of the simulation is poor (DV1 configuration problems). 

• The rule base is incomplete, so occasionally the participant will say something that is 
logical but makes no sense to the system. 

• The audio is not well synchronized to the video (DVI 2.12 limitations). 

• With the large inspection of the "procedure Options" code, there is a significant delay 
in audio file access from the CD-ROM. Tins increases as you progress with the 
inspection. 
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6.0 RECOMMENDATIONS 


6.1 Conclusion Summary 

Overall, this program appears to be effective for teaching the process of code inspection. In addition, 
analysts prefer learning with this method, or this method in combination with current methods. 


6.2 NASA Options 

Based on our conclusions, SwRI has outlined seven options for NASA to consider. The seven options 
are presented in the table below along with pros and cons for each. 


OPTIONS 

PROS 

CONS 

ESTIMATED ] 
COST TO 
IMPLEMENT 

1 . Take the Carnegie 
Mellon Course and 
use it as is (DVI 7 
board set). 

• no new software 
development 

• this course is better 
than nothing 

* DVI 7 board set is 
unavailable 

* old technology (DVI 
version) 

* as is, die simulation 
is cumbersome to 
use 

* no technical support 
for the existing 
software (Carnegie 
Mellon) 

* no technical support 
for this version of 
DVI hardware or 
software (Intel) 

none 

2. Take the Carnegie 
Mellon Course and 
modify the existing 
course (DVI 7 
board set). 

• could make minor 
changes to the data 
files of the course, 
not the source code 
(e.g., images, color, 
enlarge "hot spots”) 
(non-ins tructional 
changes) 

* DVI 7 board set is 
unavailable 

* old technology (DVI 
version) 

* as is, the simulation 
is cumbersome to 
use 

* no technical support 
for consultation 
(Carnegie Mellon) 

* no technical support 
for this version of 
DVI hardware or 
software (Intel) 

$25,000 
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3. Port and upgrade • new technology • software $75,000 

the Carnegie Mellon (higher quality development effort 

Course to a new version of (porting and 

system and modify interactive video) upgrading) 

the existing course • could make minor • no technical support 

as done in Option 2 changes to the data from Carnegie 

(current DVI files of the course, Mellon to port 

hardware). not the source code • limited technical 

(e.g., images, color, support from Intel 
enlarge "hot spots”) (regarding the old 

(non-instructional version) 

changes) • as is, die simulation 

is cumbersome to 
use 

• possible 
compatibility 
problems with the 
existing expert 
module and new 
DVI software if the 
expert module is 

simply ported 

4. Port and upgrade • new technology • software $100,000 

the Carnegie Mellon (higher quality development effort 

Course to a new version of (porting, upgr ading , 

system and modify interactive video) and modifying) 

the course design • improve quality of • no technical support 

(current DVI instructional design from Carnegie 

hardware). (e.g., more Mellon to port 

scenarios) • limited technical 

* add/change content support from Intel 

* could replace the (regarding the old 

expert module with version) 

a better one • possible 

compatibility 
problems with 
existing expert 
module and new 
DVI software or 
software 

development effort 
for new expert 
module 
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5A Create new source 
code and data files 
using this course as 
a model (current 
DVI hardware). 

Note; This option 
would incorporate 
full motion video 
scenarios in place 
of a simulation. 


SB Create new source 
code and data files 
using this course as 
a model (current 
DVI hardware). 

Note: This option 
involves creation of 
a new simulation 
(including the 
natural language 
interface). 


new technology 
(higher quality 
version of 
interactive video) 
improve quality of 
instructional design 
and tailor to NASA 
process 

add/change content 
not dependent on 
Carnegie Mellon for 
support 

use lessons learned 
from the existing 
course 

a portion of the 
instructional 
development is 
already done 
a simulation would 
not have to be 
created (including 
die natural language 
interface) 


new technology 
(higher quality 
version of 
interactive video) 
improve quality of 
instructional design 
and tailor to NASA 
process 

add/change content 
not dependent on 
Carnegie Mellon for 
support 

use lessons learned 
from the existing 
course 

a portion of the 
instructional 
development is 
already done 


major software 
development effort 
scenario 

development effort 


$300,000 


• major software 
development effort 

• simulation 
development effort 
(including the 
natural language 
interface) 


$400, 
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6. Create a totally new 

• new technology 

* most expensive 

$500,000 

course (current DVT 

(higher quality 

option 


hardware). 

version of 
interactive video) 

• improve quality of 
instructional design 
and tailor to NASA 
process 

• not dependent on 
Carnegie Mellon for 
support 

• use lessons learned 

* extensive software 
development effort 

• more instructional 
design effort 
involved 



from existing course 




6 J SwRI Recommendation 

SwRI believes this study indicates interactive video instruction combined with simulation is effective for 
teaching software processes. SwRI believes either option Five A or Five B will produce the most 
effective results. Options Five A and Five B are the same with the exception of the simulation. Both 
options involve creation of new source code and data files, but use much of the existing content and 
course design. Although both options involve a significant software development effort, many benefits 
are gained. Both options incorporate new technology which will produce higher quality audio and video. 
Content can be changed and added, and the quality of instructional design can be improved to tailor the 
course to the NASA process. The instructional development effort is minimized by modeling the existing 
course. In addition, lessons learned from the existing course can be applied to the new course. Finally, 
by creating a new course, NASA is not dependent on Carnegie Mellon for support. 

Option Five B includes creation of a new full simulation. SwRI recognizes that creation of a full 
simulation is expensive; therefore an alternative is offered in option Five A which will provide many of 
the benefits of a full simulation at a lower cost. A major strength of the simulation in the existing course 
is that it gives the user scenarios to learn from. The alternative option. Five A, gives the user access to 
the data base of the expert system; however, it becomes menu driven, making it easy for a user to access 
specific information desired. By implementing this alternative option (Five A), creation of a new natural 
language interface needed for a full simulation is also avoided. Following is a brief summary of option 
Five A: 


• Rework the instructional modules incorporating modifications as recommended in the following 
section. 

• Instead of creating a new simulation, create scenarios using full motion video. These scenarios 
could be incorporated into the instructional modules, or included as a separate part of the course. 

Scenarios from the existing course could be expanded and new scenarios could be created, eliminating 
many of the text screens contained in the existing course. Scenarios could be developed to illustrate: 

• each phase in the code inspection process 

• roles on a code inspection team 
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variables affecting a code inspection meeting (e.g., individual personalities, level of preparation) 




6.3.1 Modification Suggestions 

Both options Five A and Five B involve use of some content and design from the existing course. SwRI 
suggests the following modifications be made to existing parts used, in order to meet the needs of NASA 
analysts. 

6.3. 1.1 Content 

* state the purpose of the course 

* state the objectives of the course 

* add content to the course per recommendations (see Appendix F) 

* change content in the course per recommendations (see Appendix F) 

* add steps in the model for code inspection per recommendations (see Appendix F) 

* change order of steps in the model for code inspection per recommendations (see Appendix 
F) 

* include more full motion video scenarios to present information 

* organize content more carefully (e.g., it is confusing if the producer is discussed in the 
section on the moderator) 

* omit the section on family and social groups and make foe re maining content on groups 
relevant to code inspection 

* provide on-line, context sensitive help 

6.3. 1.2 Presentation 

* present content in smaller chunks within foe instructional modules 

* make the instructional modules more interactive (e.g., insert practice questions with feedback) 

* provide summaries/reviews at foe end of sections within foe instructional modules 

* place less text on each screen (more white space) 

* choose text and background colors which make foe instruction more readable (contrast 
between text and background) 

* support text with audio (audio should not contradict or interfere with visuals) 

* use audio consistently with each screen (or indicate there is no audio with a particular screen) 

6.3. 1.3 User Interface Features 

* provide instructions on how to use foe program (e.g., floorplan) 

* provide instructions for navigating in the program (describe buttons or make them more 
descriptive of their action) 

* provide a method for exiting foe program during introductory segments (e.g., first time 
through instructional modules, library) 

* give foe user control over when to proceed to foe next screen (consistent) 

* make provisions for foe learner to pause at any time and resume from that point 

* place the mouse on foe screen in the position where foe user is most likely to click 

* state the purpose of foe tools and when they are available to foe user 

* provide more and better instruction on how to use foe tools 
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• provide more and better instruction on how to use the natural language interface in the 
simulation 

* make the simulation respond more quickly 

In summary, SwRI recommends option Five A as the most instructionally effective and the most cost 
effective option for incorporating process simulation training into current tr ainin g efforts. 
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APPENDIX A 


INSTRUMENTS 


(PRETEST/POSTTEST) 



Completion Tune: 


Name: 

Start Time: 

CODE INSPECTION COURSE 
- Posttest - 


Part 1: Definitions 

Directions: Define each of the following terms. 

1. Define the following roles. 

A. Moderator: 

B. Reader: 

C. Recorder: 

D. Producer: 

2. Define a formal software inspection. 

3. Define a code walkthrough. 

4. Define a formal design review board. 

5. Define a task-oriented group. 


PRECEDING PAGE BLANK NOT FILMED 



Name: 


Start Time: Completion Time: 

CODE INSPECTION COURSE 


Posttest - 


Part 2: Fill in the Blank 

Directions: Answer each of the following questions. 

1. List the purposes of formal inspections. 

2. List, in order, the stages of the formal inspection process. 

3. List the benefits of the formal inspection process. 

4. List the roles that participants assume during a code inspection. 

5. List the types of checklists that can be used before or during the code inspection. 

6. List characteristics of a code inspection meeting such as length and role responsibilities. 


7. 


List basic rules for code inspection such as constraints regarding time, roles, or purposes. 



8 . 


List behavioral guidelines (for participants) that help code inspections succeed. 


9. List the functions of formal software inspections. 


10. List the functions of code walkthroughs. 


11. List the functions of formal design review boards. 


12. List advantages and disadvantages of formal software inspections. 


13. List advantages and disadvantages of code walkthroughs. 


14. List advantages and disadvantages of formal design review boards. 


IS. List techniques other than formal software inspections, code walkthroughs and formal design 
review boards for assuring software quality. 


16. List problems often faced by moderators in conducting a software inspection. 


17. List some potential problem situations emerging from interaction within the group during 
inspection. 



18. 


List some report forms used in the formal code inspection process. 


19. List common problems within a task-oriented group. 


20. List characteristics of successful task-oriented groups. 



Name: 


Start Time: Completion Time: 

CODE INSPECTION COURSE 


Posttest 


Part 3: Multiple Choice 

Directions: Circle all correct answers. For each question there may be more than one correct choice. 


1. Which of the following purposes apply to formal code inspections? 

A. to promote adherence to project style and rules of construction 

B. to promote compliance with technology practices 

C. to obtain metrics on the code producer’s performance 

D. to obtain metrics for project management and process control 


2. Which of the following formal stages are a part of die code inspection process? 

A. planning 

B. writing foe code 

C. reinspection 

D. preparation 


3. Which of the following benefits are a result of foe formal code inspection process? 

A. improves error detection 

B. integrates developer, user, and customer feedback 

C. improves productivity 

D. selects solutions to software errors 


4. Which of foe following tasks occur during foe planning stage? 

A. distribute inspection packages to participants 

B. select foe moderator 

C. select inspectors and assign roles 

D. schedule inspection meetings 



Which of the following apply to the overview stage? 

A. often led by the producer 

B. confirm schedule and receipt of materials 

C. education on code inspection 

D. background information given on work product 


Which of the following are part of the preparation stage? 

A. verifies workproduct meets entry requirements 

B. participants locate possible defects 

C. participants gain knowledge of workproduct 

D. participants brainstorm possible solutions for defects 


Do managers assume a review role during the code inspection? 

A. yes 

B. no 


Which of the following may be used as a checklist either before or during the code inspection? 

A. construction rules 

B. style guides 

C. test cases 

D. metrics checklist 


Which of the following described) a formal code inspection? 

A. small peer group process 

B. purpose is detection and correction of software product defects 

C. external review process 

D. rigorous entry and exit criteria 


Which of the following basic rules apply to code inspection? 

A. management should not be present at iiwp*ftinnn 

B. inspections are a tool for worker evaluation 

C. producers should not be the moderator of their own work 

D. inspections should be limited to approximately 2 hours 



11. Which of the following guidelines apply to successful code inspections? 

A. have at least one positive comment 

B. record all issues in public 

C. stick to technical issues 


12. Which of the following involves an external group examining die product? 

A. software inspection 

B. code walkthrough 

C. formal design review board 

13. Which of the following described) a code walkthrough? 

A. may be informal or structured 

B. method for early defect detection 

C. external process review 

D. may be large or small peer groups 

14. Which of the following could be a potential disadvantage to software inspections? 

A. focuses on producer’s perspective 

B. process stifles creativity 

C. provides early quantitative quality evaluation 

D. provides historical error database to reduce recurrences 


15. Which of the following could be a potential disadvantage to code walkthroughs? 

A. the timing of defect detection 

B. collective review of possible problems 

C. varying structure yields inconsistent results 

D. focuses on producer’s perspective 

16. Which of the following could be a potential disadvantage to formal design review boards? 

A. integrates the developer, user, and customer perspective 

B. seldom challenges the technical basis of design 

C. does not furnish management visibility for approval/disapproval of proceeding to next 
phase 

D. focuses on producer’s perspective 



17. Which of the following individuals is responsible for initiating the inspection meeting? 

A. moderator 

B. reader 

C. manager 

D. producer 


18. In the planning stage, which individual verifies with the producer that the workproduct meets 
entry criteria? 

A. moderator 

B. reader 

C. recorder 

D. manager 


19. Which of the following individuals is responsible for compiling and recording preparation times 
from the preparation logs? 

A. moderator 

B. reader 

C. recorder 

D. producer 


20. In the planning stage, the producer must provide which of the following? 

A. function descriptions 

B. comments 

C. detailed design materials 

D. support documentation 


21. During the overview stage, who is the most active participant? 

A. moderator 

B. reader 

C. recorder 

D. producer 


22. During the overview, which individual must be able to paraphrase the workproduct for other 
members? 

A. moderator 

B. reader 

C. recorder 

D. producer 



23 . 


During the overview, which individual is responsible for answering detailed questions for the 
group regarding the workproduct? 


A. 

moderator 

B. 

reader 

C. 

recorder 

D. 

producer 


24. During the inspection meeting, which individual is responsible for determining preparedness to 


continue? 

A. 

moderator 

B. 

reader 

C. 

manager 

D. 

producer 


25. During the inspection meeting, which individual introduces the team and states the purpose of the 
meeting? 

A. moderator 

B. reader 

C. recorder 

D. producer 


26. During the inspection meeting, which individual determines the disposition of the workproduct? 


A. 

moderator 

B. 

reader 

C. 

recorder 

D. 

producer 


27. During the inspection meeting, which individual paces die group? 


A. 

manager 

B. 

reader 

C. 

recorder 

D. 

producer 





28. During the inspection meeting, which individual notes location, description, class, and type of 
defect? 

A. moderator 

B. reader 

C. recorder 

D. producer 


29. Which individual is ultimately responsible for keeping the meeting around the designated length 
of time and for closing the meeting? 

A. moderator 

B. reader 

C. recorder 

D. producer 


30. During rework, which individual verifies that defect corrections are made? 

A. moderator 

B. reader 

C. recorder 

D. producer 


31. During rework, which individual is responsible for correcting defects listed on the Inspections 
Defect List? 

A. moderator 

B. reader 

C. recorder 

D. producer 


32. During followup, which individual is responsible for completing the inspection management 
report? 

A. moderator 

B. reader 

C. recorder 

D. producer 



33. During followup, which individual is responsible for consulting with the moderator to verify that 
corrections have been completed? 


A. 

moderator 

B. 

reader 

C. 

recorder 

D. 

producer 


34. Which individual is responsible for scheduling a reinspection, if necessary? 

A. moderator 

B. reader 

C. recorder 

D. producer 


35. The purpose of the preparation log is: 

A. to record how long it took the producer to write the code to be inspected 

B. to record how long each participant took to prepare for the inspection 

C. to record how long the inspection meeting lasted 

D. to record how long the moderator spent preparing for die inspection meeting 

36. The purpose of the inspection defect list is: 

A. to provide a record of points brought up during die inspection 

B. to provide a record of preparation done for the inspection 

C. to identify solutions 

D. to provide statistics about the producer's performance 

37. The purpose of the code inspection summary report is: 

A. to provide a summary of the producer’s performance 

B. to provide a record of how long the inspection meeting lasted 

C. to provide a summary of each individual’s performance in the inspection meeting 

D. to provide a compilation of defects passed on to the moderator 

38. The purpose of the management summary report is: 

A. a detailed report incorporating information from die code inspection summary report 

B. a report of the inspection meeting by the producer’s manager 

C. a report to management regarding the producer’s level of work 

D. to provide a list of defects to management for review 



39. Which of the following is not a sign of a good inspection? 

A. accurate assessment of the workproduct 

B. defects detected efficiently 

C. solutions arrived at for defects found 

D. cooperation between group members 

40. Which of the following is qq! characteristic of a task-oriented group? 

A. members are actively involved in group problem solving 

B. comes together to accomplish goal or task 

C. achieves goals through effective communication 

D. organized in an informal way 



APPENDIX A 
INSTRUMENTS 
(ANALYST QUESTIONNAIRE) 




Name: 


Date: 


Start Time: 


Completion Time: 


ANALYST QUESTIONNAIRE 


Directions: Circle the number corresponding to your opinion about the statement. Add 

comments where appropriate. 


STATEMENT 


Overall Evaluation of the Course 

1. I liked this method of instruction. 

2. I prefer this method of instruction to the 
way information is currently being taught. 

3. This course was motivational and held my 
interest. 

4. I think this course has a number of strengths 
that make it appealing. 

5. I think this course has some weaknesses that 
need to be overcome. 

6. The course was too long and involved for 
me. 

7. I think this specific course has potential for 
use at NASA. 

8. I can apply the skills I have learned in this 
course to my job. 

9. I would like to see more courses of this type 
offered by NASA. 


RATING 

Strongly No Strongly 

Disagree Opinion Agree 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 




Strongly 

Disagree 

No 

Opinion 

Strongly 

Agree 

Course Content 





1. The code inspection model used in this 
course is similar to what I currently use. 

1 2 

3 

4 

5 

2. The content of this course is important / 
relevant to me. 

1 2 

3 

4 

5 

3. The purpose of this course was clear to me. 

1 2 

3 

4 

5 

4. The course content was academically chal- 
lenging for me. 

1 2 

3 

4 

5 

5. The level of detail in this course was appro- 
priate for preparing me to participate in a 
code inspection. 

1 2 

3 

4 

5 

Learning Effectiveness 





1. I learned about code inspection from this 
course. 

1 2 

3 

4 

5 

2. I learned more from this method of instruc- 
tion than other current methods of instruc- 
tion. 

1 2 

3 

4 

5 

Instructional Presentation 





1. The course captured my attention. 

1 2 

3 

4 

5 

2. I understood what the objectives of this 
course were. 

1 2 

3 

4 

5 

3. Prerequisite skills for this course were made 
clear to the user. 

1 2 

3 

4 

5 

4. The course material was clear and well orga- 
nized. 

1 2 

3 

4 

5 

3. Overall, enough opportunity was given for 
me to practice what I learned. 

1 2 

3 

4 

5 

6. Course feedback to me was adequate. 

1 2 

3 

4 

5 



Strongly No 

Disagree Opinion 


Strongly 

Agree 


7. Course feedback to me was meaningful. 1 2 3 4 5 

8. Assessment of my performance was fair. 1 2 3 4 5 

9. Assessment of my performance was mean- 1 2 3 4 5 

ingful. 

10. Because of die method of presentation (mul- 1 2 3 4 5 

timedia, interaction, simulation, etc.), I 

believe I learned more than with current 
methods for learning code inspection. 

11. Overall, I liked the method of presentation 1 2 3 4 5 

(multimedia, interaction, simulation, etc.) 

used in this course. 

12. This type of course was appropriate for my 1 2 3 4 5 

background and experience. 


System Capabilities 

1. Overall, learning how to use the course was 1 2 3 4 5 

easy for me. 

2. Specifically, learning how to use the Simula- 1 2 3 4 5 

tion was easy for me. 

3. Remembering names and uses of commands 1 2 3 4 5 

was easy for me. 

4. I was frustrated during parts of the course. 1 2 3 4 5 

5. The course speed / response was too slow / 1 2 3 4 5 

cumbersome. 

6. Letting me control where I went added value 1 2 3 4 5 

to the instruction. 

7. Graphics were interesting and effective. 1 2 3 4 5 

8. I could understand the audio well. 1 2 3 4 5 




Strongly 

No 

Strongly 


Disagree 

Opinion 

Agree 

9. The quality of the video (full motion) was 
good. 

1 

2 

3 4 

5 

10. The video (full motion) added value to the 
course. 

1 

2 

3 4 

5 



APPENDIX A 


INSTRUMENTS 


(INTERVIEW QUESTIONS) 




Name: 


Date: 


Start Time: Completion Time: 

INTERVIEW QUESTIONS 

Overall Evaluation of the Course 

1. Did you like this method of instruction? Why or why not? 

2. Overall, what did you like most about this course? 

3. Overall, what did you like least about this course? 

4. What do you think the strengths of this course are? 

5. What do you think the weaknesses of this course are? 

Can these weaknesses be overcome? 

What are some suggestions for overcoming these weaknesses? 

6. Does this course have potential for use in teaching the code inspection course? 

7. Would you recommend this specific course be used at NASA/IBM for training analysts in the 
code inspection process? Is it better than nothing? 

What changes would you recommend? 

Why would you make these changes? 

8. Would you recommend instruction similar to this course be used by NASA/IBM to teach other 
content or processes? 



9. 


What do you think would improve this course the most? 


10. How would you suggest using this course? 

Learning Effectiveness 

1. Do you feel you learned from this course? Why or why not? 

2. If yes, what specifically did you learn? 

3. What changes would make this course more effective in teaching the content? 

Instructional Presentation 

1 . What did you like best about the method of presentation of this course? 

2. What did you like the least about the method of presentation of this course? 

3. What parts were difficult to use? Why? Be specific. 

4. How does this course compare to how you currently receive instruction in the code inspection 
process? 

Which method do you prefer for learnings 
Course Content 

1. Is the content of this course relevant to you? 

2. What content specifically is most relevant for you? 

What content specifically is most irrelevant for you? 


What content is missing that you view as relevant and should be added to the course? 



3. If you used this course as it presently exists, what parts would you use in terms of content? 
What parts, if any, would you omit in terms of content? 

Other general suggestions for use? 

4. How could this course content be changed to more closely fit NASA needs? 

5. What other content/processes do you think this type of instruction might be appropriate for? Be 
specific. 

6. How realistic were the video scenarios during the instructional modules? 

7. How realistic were the video scenarios during the simulation? 

System Capabilities 

1. Did you have difficulties with any sections of the course? 

If yes, what areas of the course did you have difficulty with? 

Do you think these difficulties could be overcome? 

What suggestions do you have for overcoming these difficulties? 

2. What is your opinion of the ease of use in the following parts of the course? 

Auditorium: 

Training Room: 

Module 1 (Formal Inspections: Purpose and Process): 


Module 2 (Inspections Types and Differences): 



Module 3 (Inspection Roles and Pitfalls): 


Module 4 (Inspections Tools and Forms): 
Module 5 (Inspections Communications): 

Library: 

Office: 

Conference Room (simulated code inspections): 



APPENDIX A 


INSTRUMENTS 


(OBSERVATION FORM) 




Name: 


Date: 


Start Time: Completion Time: 


OBSERVATION FORM 

TASK PROBLEMS/ ANALYST OBSERVER TIME 

DIFFICULTIES COMMENTS COMMENTS SPENT 

Code Inspection Course 
Auditorium 


Training Room 
Module 1 


Module 2 


Module 3 


Module 4 


Module 5 


Posttest (written) 


Code Inspection Course 
Practice Inspection 


Library 



Office 


Code Inspection 


Analyst Questionnaire 


Interview Questions 


Posttest (written) 



APPENDIX A 


INSTRUMENTS 

(DEMOGRAPHIC DATA SHEET) 



DEMOGRAPHIC INFORMATION 


Name: 

Tide: 

Degree(s): 

Computer Experience 

1 . How many years have you used computers? 

2. How many hours per day do you currently use computers in your work? 

3. For what applications do you currently use computers (programming, word processing, etc.)? 


Programming Experience 

1 . How many college level courses have you taken where you were required to write or understand 

a program in a procedural programming language such as Pascal, FORTRAN, C, Ada, etc? 


How many college-level courses have you taken where you were required to write or understand 

a program in the Ada programming language? Briefly describe your level of experience with the 
Ada programming language. 


3. 


How many computer languages can you understand and program with? 
languages. 


Please list these 


Code Inspection Process 

1- Have you ever participated in a code inspection before? If yes, how many? 

2. How long have you been involved in the code inspection process? 


3. 


Is code inspection part of your present job? 



4. When was the last time you participated in a code inspection? What roles did you perform? 


5. Have you ever received formal training in the code inspection process? If yes, please describe 
briefly. 


6. Have you taken any college level courses where software engineering concepts were taught? If 
yes, how many? Please describe briefly. 


7. Have you taken any college level courses where software technical reviews were discussed? If 
yes, how many? Please describe briefly. 

General Information 

1 . Have you ever received training via die computer before (computer-based training)? 

If yes, how many courses and for what topics? 

Do you feel like it was an effective way to learn? 

Did you like learning from computer-based training? 

What do you feel are some strengths of computer-based training? 

What do you feel are some weaknesses of computer-based training? 

2. Have you ever received any training which incorporates audio and video images? If yes, please 
describe. 


3 . 


Have you ever received any training which incorporates a process simulation, for example code 
inspection, before? If yes, what process was taught? Please describe briefly. 



APPENDIX A 
INSTRUMENTS 
(INSTRUCTIONAL PATH) 




Name: 


Date: 


Start Time: 


Completion Tune: 


NASA CODE INSPECTION INSTRUCTIONAL VALIDATION 


Demographic Data Sheet 

Pretest (written) 

Code Inspection Course 

1. Auditorium (8 minutes) 

2. Training Room 

A. Login 

B. Module 1 • Formal Inspections: Purpose and Process (14 minutes) 

C. Module 2 - Inspection Types and Differences (20 minutes) 

D. Module 3 - Inspection Roles and Pitfalls (43 minutes) 

E. Module 4 - Inspection Tools and Forms (14 minutes) 

F. Module 5 - Inspection Communications (29 minutes) 

Posttest (written) 

G. Practice Inspection - "Options* (choose moderator from office) (15 min.) 

Record Strengths/Weaknesses 

3. Library (10 minutes for orientation and initial exploration) 

4. Office (5 minutes for exploration) 

5. Training Room or Library (return occasionally) 

6. Conference Room (inspection of the "FindMaximum* code) (recorder) (45 min.) 

Record Strengths/Weaknesses 


Analyst Questionnaire 
Interview Questions 
Posttest (written) 



NASA CODE INSPECTION INSTRUCTIONAL VALIDATION 


1. "Options* (Practice Inspection) 
Strengths: 


Weaknesses: 


2. "F1nd_Maxhnum" (Final Inspection) 


Strengths: 


Weaknesses: 



APPENDIX B 


RAW DATA 


(PROGRAM FEEDBACK) 




PROGRAM 


Strengths 

good use of emotional tone 
never introduced irrelevant 
topics 


good use of emotional tone 
never introduced irrelevant 
topics 

good job stopping tangents as 
well 


never introduced irrelevant 
topics 

good job stopping tangents 
correctly expressed a minority 
opinion 

good participation 


Weaknesses 






missed the two biggest 
errors in the code 
hanged opinion incorrectly, 
perhaps due to group 
pressure 

lack of input from you 
(review inspection 
communication module) 
too passive during 
inspection 

missed the two biggest 
errors in the code 
changed opinions 
incorrectly, perhaps due to 
group pressure 
difficulty with talk interface 
too many topics left open 
without a stated final 
resolution 

missed the two biggest 
errors in the code 
too many aggressive 
comments 









APPENDIX B 
RAW DATA 

(ANALYST QUESTIONNAIRE) 




Name: 

Date: 



Start Time: 

Completion Time: 



ANALYST QUESTIONNAIRE 



Directions: Circle the number corresponding to your opinion about the 

comments where appropriate. 

statement. Add 

STATEMENT 

RATING 




Strongly 
Disagree 
1 2 

No 

Opinion 
3 4 

Strongly 

Agree 

5 


Analyst 

#1 

Analyst 

n 

Analyst 

#3 

Overall Evaluation of the Course 




1. I liked this method of instruction. 

4 

5 

4 

2. I prefer this method of instruction to the 
way information is currently being taught. 

5 

5 

4 

3. This course was motivational and held my 
interest. 

5 

4 

4 

4. I think this course has a number of strengths 
that make it appealing. 

5 

4 

5 

S. I think this course has some weaknesses that 
need to be overcome. 

5 

4 

3 

6. The course was too long and involved for 
me. 

4 


1 

7. I think this specific course has potential for 
use at NASA. 

5 

5 

2 

8. I can apply the skills I have learned in this 
course to my job. 

5 

2 

1 


9. I would like to see more courses of this type 
offered by NASA. 

Course Content 

1. The code inspection model used in this 
course is similar to what I currently use. 

2. The content of this course is important / 
relevant to me. 

3. The purpose of this course was clear to me. 

4. The course content was academically chal- 
lenging for me. 

5. The level of detail in this course was appro- 
priate for preparing me to participate in a 
code inspection. 


Learning Effectiveness 

1. I learned about code inspection from this 
course. 

2. I learned more from this method of instruc- 
tion than other current methods of instruc- 
tion. 


Instructional Presentation 

1. The course captured my attention. 

2. I understood what the objectives of this 
course were. 

3. Prerequisite skills for this course were made 
clear to the user. 

4. The course material was clear and well orga- 
nized. 


Strongly 

No 

Strongly 

Disagree 

Opinion 

Agree 

1 2 

3 4 

5 

Analyst 

Analyst 

Analyst 

#1 

#2 

03 

5 

5 

4 

4 


3 

5 

3 

4 

5 

1 

5 

4 

3 

4 

5 

4 

4 

5 

5 

4 

5 

5 

4 

4 

5 

4 


5 1 2 

4 1 3 


5 


2 


2 




Strongly 

No 

Strongly 


Disagree 

Opinion 

Agree 


1 2 

3 4 

5 


Analyst 

Analyst 

Analyst 


#1 

#2 

#3 

5. Overall, enough opportunity was given for 
me to practice what I learned. 

4 


3 

6. Course feedback to me was adequate. 

5 


3 

7. Course feedback to me was meaningful. 

4 


4 

8. Assessment of my performance was fair. 

4 


4 

9. Assessment of my performance was mean- 
ingful. 

4 


4 

10 Because of the method of presentation (mul- 
timedia, interaction, simulation, etc.), I 
believe I learned more than with current 
methods for learning code inspection. 

5 

5 

5 

1 1 Overall, I liked the method of presentation 
(multimedia, interaction, simulation, etc.) 
used in this course. 

5 

5 

5 

12 This type of course was appropriate for my 
background and experience. 

System Capabilities 

5 

4 

4 

1. Overall, learning how to use the course was 
easy for me. 

4 

2 

4 

2. Specifically, learning how to use the simula- 
tion was easy for me. 

4 

1 

4 

3. Remembering names and uses of commands 
was easy for me. 

4 

1 

3 

4. I was frustrated during parts of the course. 

4 

3 

2 

5. The course speed / response was too slow / 
cumbersome. 

4 

. 4 

2 

6. Letting me control where I went added value 
to the instruction. 

5 

4 

4 




Strongly 

No 

Strongly 


Disagree 

Opinion 

Agree 


1 2 

3 4 

5 


Analyst 

Analyst 

Analyst 


#1 

n 

#3 

7. Graphics were interesting and effective. 

5 

5 

4 

8. I could understand the audio well. 

5 

5 

4 

9. The quality of the video (full motion) was 
good. 

4 

4 

4 

10 The video (full motion) added value to the 
course. 

5 

5 

4 



APPENDIX B 
RAW DATA 

(INTERVIEW QUESTIONS) 




Name: 


Date: 


Start Time: Completion Time: 


INTERVIEW QUESTIONS 


Overall Evaluation of the Course 

1. Did you like this method of instruction? Why or why not? 
Analyst 1: Yes 

Helpful because of real examples (video) 
Interesting 

Allows you to practice (simulation) 

Access to information in the library 
Analyst 2: Yes 

Really good, definitely worth it 
Needs some fine tuning 
Lots better than manuals 
Available (come and get what need) 

Variety 

More interesting 

Uses more senses (multimedia, many methods) 
Analyst 3: Yes, better than manuals 

Audio and video with words, get more out of it 
Motion video sequence (what if s) 


2. Overall, what did you like most about this course? 
Analyst 1 : Scenarios (video) 

Analyst 2: Video examples (especially if interactive) 

Analyst 3: Video segments 


3. Overall, what did you like least about this course? 

Analyst 1: Talk too much 

Too much information on the screen 
Audio interferes with text on screen 

Analyst 2: User interface (mouse, consistency, prompt not clear) (should be obvious) 

Analyst 3: All pretty good 

Cursor moving around 
Group section 

Audio doesn’t always follow text 

4. What do you think the strengths of this course are? 

Analyst 1 : Help you to remember information for longer time because of way it is presented 



Analyst 2: See #1 

Analyst 3: Same as above 


5. What do you think the weaknesses of this course are? 

Analyst 1 : Lots of material 

No instructions on how to operate the course 
Analyst 2: Group section not in code inspection context 

Program is unclear if tools could be used outside of the course 
Waste time learning about tools you only use for course, not in reality 
Not being able to navigate to certain parts for review 
Audio competes with text at times 
User interface 
Analyst 3: Same as above 


Can these weaknesses be overcome? 
Analyst 1: Yes 

Analyst 2: Yes, definitely, no doubt 

Analyst 3: Yes 


What are some suggestions for overcoming these weaknesses? 

Analyst 1 : Break up material 

Summary/review at end of modules 

Be able to review smaller chunks of information 

Instructions on how to operate the course 

Analyst 2: Scenarios as opposed to lengthy text 

Analyst 3: Separate course on groups and include more items on effective meetings (relate 

to code inspection) 

Does this course have potential for use in teaching the code inspection course? 

Analyst 1: Yes 

Analyst 2: Yes, definitely 

Analyst 3: Yes (introduction for a new hire, reference for more experienced employees, use 

to relieve first time tensions associated with code inspection) 

Would you recommend this specific course be used at NASA/IBM for tr aining analysts in the 

code inspection process? Is it better than nothing? 

Analyst 1: No, needs modifications 

Still better as is than no course at all 

Analyst 2: Definitely better than nothing, can get something out of it 

What matters is if the content is right for NASA 

Analyst 3: Hard to say (don’t know what they do over there) 

Cost is a consideration 


What changes would you recommend? 
Analyst 1 : Change weaknesses in #5 



Minimal 

More video (helps retain information) 

User interface 

Some content presented in such a way that I didn’t retain it 

Content (better organization or structure of content, hand hold me better through 

it) 

Same as #2-5 


Why would you make these changes? 

Analyst 1 : To make class more effective 

To learn more 
Analyst 2: Ease of use 

Analyst 3: 


8. Would you recommend instruction similar to this course be used by NASA/IBM to teach other 
content or processes? 

Analyst 1: Sure 

Analyst 2: Definitely 

Analyst 3: Requirements inspections 

Level six test case review 


9. What do you think would improve this course the most? 

Analyst 1 : More scenarios, examples, and video 

Analyst 2: User interface 

Get to sections easily and just use parts 
Analyst 3: More video segments 

10. How would you suggest using this course? 

Analyst 1: 

Analyst 2: For new hires, experienced people for review, or workstation (reference) 

Analyst 3: See #6 


Learning Effectiveness 

1 . Do you feel you learned from this course? Why or why not? 

Analyst 1: Yes 

Analyst 2: Yes 

Analyst 3: Yes, when took test, couldn’t recall, but could recognize (multiple choice okay, 

still hard on parts 1 and 2) 

Too much information in too little time 

2. If yes, what specifically did you learn? 

Analyst 1 : Never knew certain things about code inspection 

Roles of people 


Analyst 2: 


Analyst 3: 



Not only learned the material, but feel like I really attended a code inspection 
(scenarios and instructional modules both) 

Analyst 2: General information, not details (the information is there, but it is overwhelming 

and hard to get to). Important how you section information and present to people 
Analyst 3: Overall "process" of code inspection 

Roles 


3. What changes would make this course more effective in teaching the content? 

Analyst 1 : Content is pretty good 

Analyst 2: Need a purpose 

Need objectives 

Provide a course description or objective for a person so they could use or not 
use (waste time) 

Analyst 3: See previous question 


Instructional Presentation 

1. What did you like best about the method of presentation of this course? 

Analyst 1: Video 

Library to access information 
Analyst 2: Video examples (retention) 

Analyst 3: Simulation (experience of being in an inspection without actually being involved 

in one) 


2. What did you like the least about the method of presentation of this course? 

Analyst 1 : Coffee room & office were redundant 

Don’t need these rooms, can practice in training room 
Analyst 2: Text screens 

Audio at times detracted from video 

Analyst 3: Sometimes introductions were too long (I am wasting my time listening to this 

person) 

Doing the course all in one day was hard 
No major complaints 

3. What parts were difficult to use? Why? Be specific. 

Analyst 1: Simulation (in general) 

Something to bring back to main menu so can proceed quickly 
Analyst 2: Tools 

Natural language interface 

Analyst 3: None really, it was pretty simple to use 

Tools and natural language interface in modules, program doesn’t tell you they 
are just for this training and when they will be used 


4. How does this course compare to how you currently receive instruction in the code inspection 



process? 

Analyst 1 : Have never received one 

On the job training 
Learned lots from this course 
Analyst 2: It is a lot better than manuals 

Suggest following this course up with a code inspection where you just observe 
and then discussion with experienced person (the 3 go together) 

Analyst 3: Better, but weigh decisions with cost of producing training for all the areas 

It’s either "on the job training" or this 
Don’t think print base works very well at all 


Which method do you prefer for learning? 

Analyst 1 : This course first and then on the job training 

Analyst 2: Use this program, observe a code inspection, discuss with experienced person 

Analyst 3: 


Course Content 

1. Is the content of this course relevant to you? 

Analyst 1: Yes 

Analyst 2: No, I don’t do code inspection but I did want to know 

Analyst 3: No 


2. What content specifically is most relevant for you? 
Analyst 1 : Role of reviewers 

Behavior guidelines 
Interpersonal communication skills 
Analyst 2: Group dynamics 

Analyst 3: 


What content specifically is most irrelevant for you? 

Analyst 1 : Different kinds of groups (family, social) 

Analyst 2: Family and social groups (most people know this). Doesn’t relate to code 

inspection (obvious, who cares) 

Analyst 3: 

What content is missing that you view as relevant and should be added to the course? 

Analyst 1 : Summary at the end of sections 

Analyst 2: Active listening (use scenarios) is really important 

More video examples (icon available if you want to see more videos) 

More information/detail on some things 
Better definitions of terms (on line glossary) 

Order of content (not missing, but put in different location) 


Analyst 3: 



3 . 


If you used this course as it presently exists, what parts would you use in terms of content? 
Analyst 1: All except those listed in the next question 

Analyst 2: Depends on application (use what need) 

Analyst 3: 


What parts, if any, would you omit in terms of content? 

Analyst 1 : Module 1 (information comes up later in other parts) 

Tools and Forms 

Analyst 2: Depends on application (use what need) 

Analyst 3: Group part (family, social) 


Other general suggestions for use? 

Analyst 1: Don’t use all at one time 

Analyst 2: Let me see what I want to see and not get bogged down 

Analyst 3: No 


How could this course content be changed to more closely fit NASA needs? 

Analyst 1: Don’t know much about how IBM does code inspection 

If were to use, would need to follow IBM guidelines 
Analyst 2: Are they teaching IBM practices 

Provide "what ifs" 

Management suggestion too stiff, not realistic 

Analyst 3: 

What other content/processes do you think this type of instruction might be appropriate for? Be 
specific. 

Analyst 1 : Communications classes 

Learning a new language 
Analyst 2: Anything 

Training in labs can use this 
Management 
Development processes 
Whole cycle 


How realistic were the video scenarios during the instructional modules? 
Analyst 1: Yes, they were realistic 

Analyst 2: Good representations 

Analyst 3: Pretty realistic, but don’t like attacks on producer 

How realistic were the video scenarios during the simulation? 

Analyst 1: Yes, they were realistic 

Analyst 2: Never been in one, but liked the idea 

Analyst 3: Okay 



System Capabilities 


1. Did you have difficulties with any sections of the course? 

Analyst 1: Pretty easy to use 

Some problem with tools, but maybe didn’t pay enough attention to module 
Enough instruction in how to use tools 
Natural language interface 
Analyst 2: Natural language interface 

Not consistent 
Mouse 

Should be easy and not distracting 

Analyst 3: Occasional minor occurrence of what to do or click on next 

No preference using either the mouse or keyboard 

If yes, what areas of the course did you have difficulty with? 

Analyst 1: 

Analyst 2: 

Analyst 3: 


Do you think these difficulties could be overcome? 
Analyst 1: 

Analyst 2: Yes 

Analyst 3: 


What suggestions do you have for overcoming these difficulties? 

Analyst 1 : Help button (use video and audio to provide instruction) rather than text 

Analyst 2: 

Analyst 3: 


What is your opinion of the ease of use in the following parts of the course? 
Analyst 1: 

Analyst 2: 

Analyst 3: 

Auditorium: 

Analyst 1 : Very good introduction of course 

Analyst 2: 

Analyst 3: 

Training Room: 

Analyst 1: Some modules got a little long 

Analyst 2: Would like to be able to pause and back out 

Needs better initial instruction in how to navigate 



Define buttons and use consistently 

Mouse interface (make hot spot bigger, position on location to click) 
Analyst 3: Easy 

Some waiting for audio, but not a major problem 

Module 1 (Formal Inspections: Purpose and Process): 

Analyst 1: 

Analyst 2: 

Analyst 3: 


Module 2 (Inspections Types and Differences): 
Analyst 1: 

Analyst 2: 

Analyst 3: 


Module 3 (Inspection Roles and Pitfalls): 
Analyst 1: Long 

Analyst 2: 

Analyst 3: 


Module 4 (Inspections Tools and Forms): 
Analyst 1: 

Analyst 2: 

Analyst 3: 


Module 5 (Inspections Communications): 
Analyst 1 : 

Analyst 2: 

Analyst 3: 


Library: 
Analyst 1: 

Good to use 

Analyst 2: 

Good way to learn and get information 
Good, has lots of potential 

Analyst 3: 

Easy to use 

Would like a sort function to find things 
Easy to use 

Office: 
Analyst 1: 
Analyst 2: 
Analyst 3: 

Easy to use 


Liked tools 
Tools were good 






Conference Room (simulated code inspections): 


Analyst 1: 

Not very much interaction (but not used to tools or code) 
Provide chances to interact 
Was short 


Analyst 2: 
Analyst 3: 

No real problem 
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DEMOGRAPHIC INFORMATION 


Title: Analyst 1 : Associate Programmer 

Analyst 2: SSW Engineer 
Analyst 3: Software Engineer 

Degree(s): Analyst 1 : Computer Science 

Analyst 2: BS Mechanical Engineering 
Analyst 3: BS Electrical Engineering 

Computer Experience 

1. How many years have you used computers? 

Analyst 1: S years 

Analyst 2: 12 years (PC’s for basic technical computations and word processing) 

Analyst 3: 14 years 


2. How many hours per day do you currently use computers in your work? 
Analyst 1 : 6 hours 

Analyst 2: 1-2 hours per day 

Analyst 3: 4 hours 


3. For what applications do you currently use computers (programming, word processing, etc.)? 
Analyst 1: 

Analyst 2: Word processing/documentation, PROFS, storyboard graphics presentations 

Analyst 3: Word processing, EMail, graphics, information system 


Programming Experience 

1. How many college level courses have you taken where you were required to write or understand 

a program in a procedural progr amming language such as Pascal, FORTRAN, C, Ada, etc? 
Analyst 1: 7 classes 

Analyst 2: 5-6 (Fortran and Pascal) 

Analyst 3: Fortran 

2. How many college-level courses have you taken where you were required to write or understand 
a program in the Ada programming language? Briefly describe your level of experience with the 
Ada programming language. 

Analyst 1: None 

Analyst 2: None 

Analyst 3: No college level course, one week course (40 hours) on Ada 



3 . 


How many computer languages can you understand and program with? Please list these 
languages. 

Analyst 1: Pascal, C, Cobol, LSP 

Analyst 2: Basic, Fortran, Assembler (basic programs) 

Analyst 3: Fortran, Basic, C, Ada (weak in C & Ada) 


Code Inspection Process 

1 . Have you ever participated in a code inspection before? If yes, how many? 

Analyst 1: 4 

Analyst 2: No 

Analyst 3: No 


2. How long have you been involved in the code inspection process? 
Analyst 1: 2 months 

Analyst 2: 

Analyst 3: 0 


3. Is code inspection part of your present job? 

Analyst 1: Yes 

Analyst 2: Only when I need to look at code to troubleshoot a lab problem 

Analyst 3: No 


When was the last time you participated in a code inspection? What roles did you perform? 
Analyst 1: Spring 1992 (optional attendee: code review) 

Analyst 2: Never 

Analyst 3: Never 


5. Have you ever received formal training in the code inspection process? If yes, please describe 
briefly. 

Analyst 1: No 

Analyst 2: No 

Analyst 3: No 

6. Have you taken any college level courses where software engineering concepts were taught? If 
yes, how many? Please describe briefly. 

Analyst 1 : 1 class. There is one project for the whole class. The project is broken up into 

small parts so that each group is responsible for it. 

Analyst 2: No 

Analyst 3: No, only courses offered at work 



7 . 


Have you taken any college level courses where software technical reviews were discussed? If 
yes, how many? Please describe briefly. 

Analyst 1: Software engineering, made presentations on design and code implementation 

Analyst 2: No 

Analyst 3: No 


General Information 

1. Have you ever received training via the computer before (computer-based training)? 
Analyst 1: Yes 

Analyst 2: Yes 

Analyst 3: Yes 


If yes, how many courses and for what topics? 

Analyst 1: 1 course (Flight Control, GNC) 

Analyst 2: Situational Leadership, Intro to Assembler 

Analyst 3: 8 courses (GPC Synchronization, Bus Reconfiguration, PASS ILoad Recon, 

Ascent Overview, GN&C, Crew SW Interface, CRT Display Overview, PASS 
Architecture) 


Do you feel like it was an effective way to learn? 
Analyst 1: Yes 

Analyst 2: Yes 

Analyst 3: It was OK 


Did you like learning from computer-based training? 
Analyst 1: Yes 

Analyst 2: Yes 

Analyst 3: It was OK 


What do you feel are some strengths of computer-based training? 

Analyst 1 : Interactive, easy to review, look up terminology easy, know where you are and 

test helps to reinforce the ideas 

Analyst 2: Graphics capability, flexible to personal schedule, multimedia tools can be used 

Interactive sessions are great 

Analyst 3: Not as boring as reading from a manual 

More effective examples can be provided 

What do you feel are some weaknesses of computer-based training? 

Analyst 1: Slow 

Analyst 2: They are only as good as the programmer makes it. The programmer needs to 

clearly answer the key issues and questions. It can be limiting. 

Analyst 3: Nobody to answer you questions 



2 . 


Have you ever received any training which incorporates audio and video images? If yes, please 
describe. 

Analyst 1: No 

Analyst 2 : Yes. The Situational Leadership class was interactive and very effective. I have 

also seen some multimedia Shuttle presentations and am trying to develop some 
training stories on Storyboard. 

Analyst 3: No 


3. Have you ever received any training which incorporates a process simulation, for example code 
inspection, before? If yes, what process was taught? Please describe briefly. 

Analyst 1: No 

Analyst 2: No 

Analyst 3: No 



APPENDIX C 


ANNOTATED COURSE OBJECTIVES 




OBJECTIVES FOR THE CODE INSPECTION COURSE 
Advanced Learning Technologies Project 
Software Engineering Institute 
Carnegie Mellon University 


All items in bold were provided as objectives by Carnegie Mellon University; other information was 
added by SwRI after reviewing the program. 


TRAINING ROOM OBJECTIVES 
Overall Training Objective: 

To provide skills and knowledge that software engineers will need to conduct successful software 
inspections within the interactive system that will transfer into real-life inspection environments, to 
efficiently use the resources and tools within the DVI system, and to experience ease in using the system 
for learning enjoyment. 

Module 1 Objectives - Formal Inspections: Purpose and Process 
The Software Engineer will be able to: 

1. Describe the purpose of formal inspections. 

A. Set a standard of excellence 

B. Promote correctness and completeness 

C. Promote adherence to project style and rules of construction 

D. Promote compliance with technology practices 

E. Provide structured ways to view product systematically 

F. Obtain metrics for project management and process control 

2. Identify the stages of the formal inspection process. 

A. Planning (assigning tasks) 

B. Overview (communications/education) 

C. Preparation (education) 

D. Inspection (find errors) 

E. Reporting (report errors) 

F. Rework (fix errors) 

G. Follow-Up (ensure correct fixes) 

H. Reinspect (find final errors) 

3. Describe the benefits of the formal inspection process. 

A. Cost savings 

B. Improve error detection 

C. Reduce cost to customer 

D. Improve productivity 

E. Increase product knowledge 

F. Improve process control 



4. Describe the role or planning and preparation in the inspection process. 

A. Planning 

- workproduct meets entry criteria 

- moderator selected 

- decision on overview 

- inspectors selected and assigned roles 

- overview and inspection meetings scheduled 

B. Overview Session 

- often led by producer 

- educational for team 

- background information given on work product 

C. Preparation 

- workproduct must be thoroughly reviewed prior to inspection meeting 

- individual preparation to locate possible defects and gain knowledge of workproduct 

5. List the review roles assumed during the inspection. 

A. Moderator 

B. Reader 

C. Producer 

D. Recorder 

6. List the types of checklists that can be used before, during the inspection and for followup. 

A. Rules of Construction 

B. Correctness Checklist 

C. Style Checklist 

D. Metrics Checklist 

E. Technology Checklist 

* Additional Information 
Definition of an inspection: 

- a small peer group process whose purpose is the detection and correction of software product defects 

- rigorous entry and exit criteria 

- a formal procedure for identification, report and rework of workproduct defects 

Characteristics of an inspection meeting: 

- no more than 2 hours 

- initiated by moderator 

- preparation times recorded 

- reader guides the group 

- producer helps identify defects 

- recorder records defects on Inspection Defect Log 

Basic rules for Code Inspections: 

- management should not be present at inspections 

- inspections are not a tool for worker evaluation 

- producers should not be moderator, reader, or recorder on their own work 

- checklists of questions can be used to define die task and stimula te defect finding 

- inspections should be limited to 2 hours 

■ producers should not spend more than 25 % of their time in inspection-related duties 



Guidelines for Successful Code Inspections: 

- be prepared 

- be willing to associate and communicate 

- have at least one positive comment 

- find defects, not solutions 

- stick to the standard or change it 

- do not use derogatory language 

- record all issues in public 

- evaluate the product, not the producer 

- stick to the technical issues 

- keep accurate statistics 



Module 2 Objectives - Inspection Types and Differences 
The Software Engineer will be able to: 

1. Discriminate between inspections, walkthroughs and design reviews/audits. 

- Software Inspection: small group process whose purpose is the detection & correction of 

software product defects. 

- rigorous entry & exit criteria 

- process management tool for improving quality 

- Internal Walkthroughs: a dynamic presentation of a software product usually presented by the 

developer of the software to a peer group for the purpose of improving the quality of the 
work product. 

- vary in format from very informal to structured reviews 

- Formal Design Reviews/ Audits: an agent external to die process being examined. 

- insures proper validation 

- insures that producing intended results 

2. Describe the different functions of formal inspections, walkthroughs and design reviews/audits. 

- Software Inspection: 

- small trained peer group 

- specific formal agenda 

- specific roles 

- function to identify, classify, & report defects 

- process control tool 

- rigorous entry/exit product criteria 

- product examined at defined checkpoints 

- Internal Walkthroughs: 

• large or small peer groups 

- informal to structured 

- early defect detection 

- educational support 

- Formal Design Reviews/ Audits: 

- external process review 

- customer, user, & developer usually involved 

- affirms or negates status of product 

- not used for defect detection 

3. Compare the differences between inspection and walkthrough procedures. 

- Software Inspection 

- Advantages 

- formality yields consistent results 

- early quantitative quality evaluation 

- historical error database to reduce recurrences 

- Disadvantages 

- keyed to developer’s viewpoint 

- Internal Walkthroughs 

- Advantages 

- collective review and detection of defects 

- early detection of defects 


- Disadvantages 

- varying rigor yields inconsistent results 

- little 'corporate memory" to reduce recurrences 
- Formal Design Reviews/Audits 

- Advantages 

- integrate developer, user, and customer views 

- furnish management visibility for approval or disapproval of proceeding to next phase 

- Disadvantages 

- seldom challenge technical basis of design 

- not effective for quality evaluations 

4. List other techniques for assuring software quality. 

A. Automatic Tools Checking 

B. Team Leader Checking 

C. Simulation 

D. Prototyping 

E. Unit, Subsystem, and System Testing 



Module 3 Objectives - Inspection Roles and Pitfalls 
The Software Engineer will be able to: 

1. Describe the roles of moderator, reader, recorder and producer in formal inspections. 

A. Moderator 

- responsible for verifying entry and exit criteria 

- making sure everyone is prepared and contributing 

makin g sure reviewers do not go off on tangents during die inspection 

- making sure that the focus of the inspection remains on the code and not the producer 

B. Reader 

- responsible for letting everyone know what is being discussed 

- pacing the meeting 

- introducing and summarizing the next piece of code to be discussed 

C. Recorder 

- responsible for writing down defects found during the inspec t io n 

- classifying errors according to predefined categories 

- noting action items to take care of after the review is complete 

D. Producer 

- writer of the code being inspected 

- answer any specific questions about the code 

- present information about the code without getting defensive when the code is questioned for 

defects 

2. Describe special problems often faced by moderators in conducting a software inspection. 

A. Attack on producer 

B. Moderator do minate 

C. Followup communications 

D. Pitfalls 

3. Describe in detail the roles of the manager, moderator, reader, recorder and producer at each 
step in the formalized inspection process. 

A. Planning 

1. Manager 

- involved 

2. Moderator 

- selected from unrelated project by producer or first line manager 

- verifies with producer that workproduct meets entry criteria 

3. Reader 

4. Recorder 

5. Producer 

- entry criteria 

- clean compile/assembly with time ta gs 

- functions descriptions and comments 

- detailed design materials 

- change request (if appropriate) 

- support documentation 



B. Overview 

1. Manager 

2. Moderator 

- schedules meetings 

- makes physical arrangements 

- sends notice of meeting time and place 

- makes sure all members get materials needed for preparation 

- gets confirmation of members’ acceptance of schedule and receipt of materials 

3. Reader 

- must be familiar with workproduct so as to be able to paraphrase the workproduct in detail 

4. Recorder 

- education regarding workproduct 
• learn classification of defects 

5. Producer 

- assemble pertinent documentation 

• presents and educates group regarding workproduct 
- producer provides tutorial on specialized design or implementation technique 

C. Preparation 

1. Manager 

2. Moderator 

- study workproduct 

- note defects 

3. Reader 

- study the workproduct and specifications documents 

- organize a strategy for paraphrasing the workproduct 

4. Recorder 

- study the workproduct 

- identify defects 

5. Producer 

-review workproduct prior to inspection meeting 

D. Inspection Meeting 

1. Manager 

2. Moderator 

- introduces team 

- states purpose of the meeting 

- checks for changes in baseline 

- checks all materials provided 

- records preparation times 

- determines preparedness to continue 

- keeps group on target and meeting objectives 

- determines disposition of workproduct 

3. Reader 

- paces the group and guides the group by paraphrasing the code 

- keeps track of location of issues and refocuses group on relevant parts of product 

- reader knows workproduct and paraphrases segments 

4. Recorder 

- notes location, description, class, and type of defect 

- recorder needs technical awareness of workproduct 

- needs good judgment and ability to classify defects 

- all issues must be recorded completely and accurately 


5. Producer 

• participates as a reviewer and raises issues about the workproduct 

- acts as an inspector and identifies defects 

- adopts non-defensive attitude 

E. Reporting 

1. Manager 

2. Moderator 

- closes inspection meeting (if pass) 

3. Reader 

4. Recorder 

- all issues must be recorded completely and accurately 

- fills out inspection defect list 

5. Producer 

F. Rework 

1. Manager 

- involved 

2. Moderator 

- verifies defect corrections made 

3. Reader 

4. Recorder 

5. Producer 

• performs rework 

- corrects defects listed on the Inspections Defect List 

- producer along with moderator helps resolve open issues 

G. Followup 

1. Manager 

2. Moderator 

- completes inspection management report 

3. Reader 

4. Recorder 

5. Producer 

- producer consults with moderator to verify that corrections have been completed 

- moderator handles reporting to management that corrections are complete 

H. Reinspection 

1. Manager 

2. Moderator 

- schedules reinspection (same as rescheduling the initial inspection) 

• completes physical arrangement 

- sends notice time and place 

- provides materials 

- inspectors confirm schedules 

3. Reader 

- attends when scheduled 

4. Recorder 

- attends when scheduled 

5. Producer 

- participates as an inspector of the workproduct 

- producer helps locate final errors 



4. Identify the checklists and forms used by each participant in the formal inspection process. 

Forms: 

A. Preparation Log 

- completed by all inspectors and given to the moderator as a record of the preparation done for 

the inspection 

B. Inspection Defect List 

- completed by the recorder during the inspection as a record of points brought up during review 

C. Code Inspection Summary Report 

- completed by the recorder following the inspection from data collected in the inspection defect 

list; passed on to foe moderator when finished 

D. Management Summary Report 

- completed by foe moderator following foe inspection, incorporating information from foe Code 

Inspection Summary Report; given to management when finished 

5. Discriminate among potential problem situations emerging from interaction within the group 
during inspection. 

Helpful Hints (moderator) (What if): 

- a quiet person hasn’t spoken yet? 

- someone talks too much? 

- someone is too aggressive? 

- everyone isn’t prepared? 

- someone tries to rush through foe inspection? 

- someone has really been obstructive during foe inspection? 

- foe meeting drifts into irrelevant subjects or unnecessary detail? 

- you haven’t assembled foe materials needed for foe inspection team? 

- foe product being inspected isn’t very good? 

- a good inspection wasn’t obtained? 

- etc. (4 screens) 

6. Identify with the "model behavior” of each participant in the inspection. 

Key Responsibilities: 

A. Manager 

- Pl annin g 

- Rework 

B. Moderator 

- Overview 

- Preparation 

- Code Inspection Meeting 

- Reporting 

- Follow-up 

- Reinspection 

C. Reader 

- Overview 

- Preparation 

- Code Inspection Meeting 

- Reinspection 

D. Recorder 

- Overview 

- Preparation 

- Code Inspection Meeting 



• Reporting 

- Reinspection 
E. Producer 

- Overview 

- Preparation 

- Code Inspection Meeting 

- Rework 

- Follow-up 

- Reinspection 

* Additional Information 
A Good Inspection: 

- accurate assessment of the workproduct 

- defects detected efficiently 

Disposition Categories: 

- Pass (meets exit criteria) 

- Does Not Pass (rework, reinspect) 

Defect: 

- non-compliance with a product specification or document standard 
• defect classes (Fagan) 

M - Missing (material called for in specs, but not included) 

E - Extra (exceeds specifications) 

W - Wrong (material is present, but contains flaw) 

- generic set of defect classes 

DE - design error 
IN - interface 
DA - data 
LO - logic 
PF - performance 
10 - input/output 
CC - code comment 
ST - standards 
DC - documentation 
SN - syntax 



Module 4 Objectives - Inspection Tools and Forms 
The Software Engineer will be able to: 


1. Demonstrate the use of the computer tools (debugger, code analysis, and notetaking) for 
analyzing a piece of Ada code. 

Code Inspection Tools: 

- hypertext tools 

- help consistency checking between documents 
• keep track of notes 

- code debugger tools 

- help check correctness of ADA code 

- enable better understanding of the code to be inspected 

2. Prepare for the simulated code inspection by analyzing and taking notes on an Ada code sample. 

3. Describe how checklists and report forms are used before, during, and in follow-up to the 
formal inspection process. 

Checklists: 

- rules of construction 

- correctness checklist 

- style checklist 

- metrics checklist 

- technology checklist 
Report Forms: 

- preparation log 

- inspection defect list 

- code inspection summary report 

- management summary report 

4. Demonstrate use of the recording form and summary form. 

5. Describe the process for notifying team members about the review when assuming the role of 
moderator. 

- schedules meetings 

- makes physical arrangements 

- sends notice of meeting time and place 

- makes sure all members get materials needed for preparation 

- gets confirmation of members' acceptance of schedule and receipt of materials 

6. Describe the process for accessing the computer tools within the "office" environment at Unimex. 

- use tools in office and training room 

- pop-up menu (left mouse button) 

- move/resize window (right mouse button) 

* Additional Objectives (from Carnegie-Mellon) 

7. Appreciate the value of software inspections as an effective technique for improving software 
quality. 



8. Acknowledge the importance of inspecting software as an organizational approach to cost 
reduction and improved productivity. 

9. Value the impact of controlling for software techniques. 



Module 5 Objectives - Inspection Communications 
The Software Engineer will be able to: 

1. Demonstrate the use of the interface tool for communicating with the simulated inspections 
group. 

Task Interface Summary (Conversational Interface) 

Purpose: to create a sentence to say to the other reviewers during a code inspection 
Consists of: 

- talk menus - sentence window 

- code window - emotions icons 

- specifications window - reviewers icon 

- notes window 

- To enter the talk interface, click a mouse button while another reviewer is talking. You will given 

a prompt that you will be given the floor shortly. 

* To exit the talk interface, click the mouse on the image of the reviewers. 

- The code being inspected, specifications for this code, and your notes about the code are accessible 

via text windows in the talk interface. 

- Remember that if you need assistance while in the talk interface, you can always access the help 

icon. 

2. Show how the attitudinal attributes (icons and phrases) are selected and used for effective 
emotional context in communication with the simulated group. 

Icons: blue - yellow (neutral) - red 
Phrases: carry attributes 

3. Recognize the existence of group experience within his/her own life pattern. 

4. Accept the importance of group dynamics as a human interactive communications skill. 

5. Define what a group is and why we function as groups. 

- its membership can be defined 

- it possesses a group consciousness 

- it possesses a shared sense of purpose 

- its members have an interdependence in the satisfaction of their nerd s 

* interaction among the members is evident and the group is able to act in a unitin g manner 

- interaction and communications are necessary in order to reach the shared goal, sod decisio ns must 

be agreed to by at least a majority of the members of the group 

6. Identify common problems within a task-oriented group. 

- 'detrimental conflict” 

- types of members 

- aggressive 

- silent 

- abusive 

- rambling 

- snapping (witty) 



7. Discriminate between a social group and a task-oriented group. 

Types of Groups: 

- social 

• organized in an informal way 

- main purpose is social in nature 

- family 

- nucleus for learning, love, trust, intimacy, acceptance and self-worth 

- forms the basis for behaviors 

- task-oriented 

- achieves goals through effective communication 

- participants more likely to accept decision results because they were actively involved in the 

group problem solving 

- comes together to accomplish goal or task 

- heart of any effective organization 

8. Identify the characteristics of a successful task-oriented group meeting and how these same 
characteristics apply to the formalized software inspection. 

- company policy 

- meeting has structure and agenda 

- preparation done prior to meeting 

- effective interpersonal communication 

- no interruptions 

- focus for conclusions and follow-up 

9. Recognize the non-verbal and verbal messages which signal problems within a group meeting. 

10. Use group process skills to effectively communicate with the simulated members of the 
inspections group. 


* Additional Objectives (from Carnegie-Mellon) 

11. Examine the positive and negative aspects of his/her participation in the simulated inspection, 
and determine the aspects of his/her role which influenced the inspection outcome, through 
review of the feedback presented during the inspection simulation as well as from the follow-up 
progress report. 



Manager Track Objectives 
The Manager will be able to: 

1. Identify the role of the manager in the inspection process. 

2. Describe the purpose of formal inspections. 

3. Identify the stages of the formal inspection process. 

- Module 1, Question 2 

4. Cite the advantages of inspection versus walkthroughs. 

- Module 2, Question 3 

5. Describe the way in which inspection data can be used in future software development 
planning. 

6 . Identify the key features of formal inspections that contribute to cost savings and error 
reduction. 

- Module 1, Question 3, etc. 

7. Cite the studies that support the use of formal inspections within organizations. 

- see library articles 

8. Describe a process for implementing formal inspections within an organization. 

9. Describe the key philosophical aspects of implementing inspections within an organization. 


System Use Objectives (from Carnegie Mellon) 

1. Independently access all resources within the code inspection course, including the library, 
training room, and context-sensitive help system. 

2. Effectively analyze the code and type in defects ideas into the notes window, for later retrieval 
during the inspection simulation. 




APPENDIX D 


CARNEGIE MELLON COURSE OBJECTIVES 




Objectives for the Code inspection Course 


Advanced Learning Technologies Project 
Software Engineering Institute 
Carnegie Mellon University 


inspection Learning Objectives: 

The software engineer will be able to: 

1. Describe the purpose of formal inspections. 

2. Identity the stages of the formal inspection prooess. 

3. Describe the benefits of the forma! Inspection prooess. 

4. Describe the role of planning and preparation in the inspection process. 

5. List the review roles assumed during the inspection. 

6. Ust types of checfcBsts that osn be used before, during, and after the inspection. 

7. Discriminate be t ween inspecti ons , walkthroughs, and design reviews/audHs. 

8. Describe the different functions of format Im p act ions . waMhroughs, and design 
reviews/audits. 

8. Compare the process deferences between inspections and walkthroughs. 

10. Describe the roles of moderator, reader, reoordtr, and produoer in formal 
inspections. 

1 1 . Describe the special problems often faced by moderators in oonduding a software 
inspection. 

12. Describe in detail the role of the manager, moderator, reader, reoordtr, and 
producer at each step in the formalized Ins pection process. 

13. identity the checklists and forms ustd by each p ar tici pa n t in the formal inspection 
process. 

14. Appreciate the value of software inspection s as an effective technique for 
improving software quality. 

15. Acknowledge the I mp or t an ce of inspecting software as an organizational 
approach to oost reduction and Improved productivity. 

1 6. Value the impact of controlling for software defects. 

17. Understand the importance of group dynamics as. a human interactive 
communication skfiL 

18. Define what a group is and why we function as groups. 

19. identity common problems within a task-oriented group. 

20. Discriminate between asocial group and a task-orisntedgroup. 



21. Identify the characteristics of a successful task-oriented group meetlno and how 
these same characteristics apply to the formalized software 

22. Recognize the nonverbal and verbal messages which signal problems within a 
group meeting. 

23 ' 5*^5! m# POk^end negative aspects of his/her participation in the simulated 
inspection, and determine the aspects of his/her role which Influenced the 
inspection outcome, through, review of the feedback presented during the 
inspedion simulation as well ae from the follow-up progress report 0 


System Uee Objectives: 

The software engineer wfll be able to: 


1. Independently a cc ess so resources within the code Inspection course. Including 

the library, training room, and co n t ext s ens itive help system. 00,00 

2. Demonstrate skill using the source level debugging and hypartaxt tools in the 
office environment for analyzing code samples prior to the inspedtonof that code? 

4 ‘ ** 1001 ^ communicating with the simulated 

'oSS? 8 ““ - *• whw, 



TRAINING ROOM OBJECTIVES 


Training Room Modules Objectives Draft #3 

April 7, 1989 


Over All Training Objective 

To provide skills and knowledge that software engineers will need to 
conduct successful software inspections within the interactive system 
that will transfer into real life inspections environments, to efficiently 
use the resources and tools within the DVI system and to experience ease 
in using the system for learning enjoyment 


Module 1 Objectives — Formal Inspections: Purpose and Process 

The Software Engineer will be able to: 

1 . Describe the purpose of formal inspections 

2. Identify the stages of the formal inspection process 

3. Describe the benefits of the formal inspection process 

4. Describe the role of planning and preparation in the inspection process 

5. List the review roles assumed during the inspection 

6. List types of checklists that can be used before, during the inspection 
and for follow-up 


Training Room Modules Objectives Draft #3 

April 7, 1989 


Module 2 Objectives - Inspections Types and Differences 

The Software Engineer will be able to: 

1 . Discriminate between inspections, walkthroughs and desion 
reviews/audits 

2. Describe the different functions of formal inspections, walkthroughs 
and design reviews/audits 

3. Compare the differences between inspection and walkthrough 

procedures * 

4. List other techniques for assuring software quality 



Training Room Modules Objectives Draft #3 

April 7, 1989 


Module 3 Objectives — Inspection Roles and Pitfalls 


The Software Engineer will be able to : 

1 . Describe the roles of moderator, reader, recorder and producer in 

formal inspections 

2. Describe special problems often faced by moderators in conducting a 
software inspection 

3. Describe in detail the role of the manager, moderator, reader, recorder 
and producer at each step in the formalized inspection process 

4. Identify the checklists and forms used by each participant in the 
formal inspection process 

5. Discriminate among potential problem situations emerging from 
interaction within the group during inspection 

6. Identify with the "model behavior" of each participant in the 
inspection 



Training Room Modules Objectives Draft #3 

April 7, 1 989 


Module 4 Objectives — Inspections Tools and Forms 


The Software Engineer will be able to: 

1 D9 ~ 9 ' W use 0f ,he computer tools (debugger, code analysis, 

and notetaking) for analyzing a piece of Ada code 

2. Prepare for the simulated code inspection by analyzing and taking 
notes on an Ada code sample * 

3 in f T * 66 h0 ^ C u heCk ,lsts . and report forms are used before, during, and 
in follow-up to the formal inspections process 

4 . Demonstrate use of the recording form and summary form 

5. Describe the process for notifying team members about the review 
when assuming the role of moderator 

6 thG process for acc essing the computer tools within the 

office environment at Ultimex 



Training Room Modules Objectives Draft #3 

April 7, 1989 


Module 5 Objectives — Inspections Communications 


The Software Engineer will be able to: 

1 . Demonstrate the use of the interface tool for communicating with the 
simulated inspections group 

2. Show how the attitudinal attributes (icons and phrases) are selected 
and used for effective emotional context in communication with the 
simulated group. 

3. Recognize the existence of group experience within his/her own life 
pattern 

4. Accept the importance of group dynamics as a human interactive 
communications skill 

5. Define what a group is and why we function as groups 

6. Identify common problems within a task-oriented group 

7 . Discriminate between a social group and a task-oriented group 

8. Identify the characteristics of a successful task-oriented group 
meeting and how these same characteristics apply to the formalized 
software inspection 

9. Recognize the non-verbal and verbal messages which signal problems 
within a group meeting 

10. Use group process skills to effectively communicate with the 
simulated members of the inspections group 



Training Room Modules Objectives Draft #3 

April 7, 1989 


Manager Track Objectives 
The Manager will be able to: 


1 . Identify the role of the manager in the Inspections Process 

2. Describe the purpose of formal inspections 

3. Identify the stages of the formal Inspection process 

4. Cite the advantages of Inspections versus Walkthroughs 

5. Describe the way in which Inspections data can be used in future 
software development planning 

6. Identify the key features of formal inspections that contribute to cost 
saving and error reduction 

orgartLtonT** 65 SUPPOrt the US * 0f tonnal within 


o rgan iration * ProC8SS for im P lementln 3 formal inspections within 


an 


the ^ Philosophical aspects of implementing inspections 
within an organization. K 



APPENDIX E 


CARNEGIE MELLON CODE INSPECTION MODEL 




Coae inspections Moaeis 
Advanced Learning Technologies Simulation 
The Inspections Process 

Planning 

Preparation 

Entry Criteria 

Conduct 

Exit Criteria 

Reporting 

Foliow-up 

Defined Roles In the Simulation 

Differences described in roles between the ALT project and other models 
are primarily due to the interface issues 

Manager - receives reports, manages follow-up 

Producer - produces product, satisfies entry criteria, explains product 
contributes as inspector, reworks product 

Moderator - verifies entry criteria via E-mail, schedules meeting via 
E-mail, provides meeting notice and materials (checklists) via E-mail to 
team, conducts overview, prepares for review as any other inspector, 
directs inspection, handles final disposition of the meeting, completes 
summary report, sends report to manager via E-mail 

Recorder - prepares for review as any other inspector, records defects, 
records issues, raises issues, provides defect log to moderator 

Reader - prepares for review as any other inspector, guides team by 
pacing the examination of the material, paraphrases material, raises 
issues 

Reviewer (All 4 people listed above - producer, moderator, 
recorder, reader) • responsible for effective participation 

Review Sequencing 

The code provides the main mechanism for sequencing the review 
discussion. Checklists will be available for the preparation process. 

Entry and exit criteria will be checked during the simulation. Recording 
logs and summary reports will be constructed as templates. 



APPENDIX F 


IBM COMPARISON OF OBJECTIVES AND CODE INSPECTION MODELS 
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r r o iT* : r, r- — H 0 U 7 M 5 C C 

To: WFRLJETT — VMSPFHQU 


Date and time 


^RGM: JIM GRR , EXT849 1 

*** Hw = p ri riing note of 34/10/92 14:^6 

Bei = w J * the response to the two actions due by mid April. if there is 
any additional information that is needed, please contact me. 

phc-j.iC^i b e out o+ the office, please reel free to use the BEE-'E^. E 37 - 5 - y 7 
tu ~each me without p laying "phone tag", w 

Subject: Reply To Action On Instructional Tool For Inspections 

^ 1 1 ^ /st-et i ng with FuSD, Southwest Research Institute, 
demo tool developed by Carnegie Mellon using 

DARP A f u n d s * 

~ ~ P — * -his memo completes the follow! no: 

1 1 > Evaluate the Carnegie Mellon Objectives 

(2) — valuate the IBM inspection process model described by 
Carnegie Mellon 

ZlllZlI " ^srnegie Meilon Objectives For The Code Inspection Course 


Additional items need to be added under "Describe the purpose of 
r • Pipewtions". i hese additional items are: 

Inspections are performeo with the expectation that all errors founts 
2 S correct f d bs<f °re they are propagated to further products 
yy produce a used for process improvement (in addition t o 
Project management and process control ) . 

Acdioional items need to be added under "Identify the stages of fo-m*’- 
i -Spec. ion process. These additional stages are: 

tntry, Criteria (prior to Planning). Inspections must satisfy a 
2~~.~ r TSasur sbx e actions that must be complete before each type 
inspect ion c^n bake place, 

^meeting Error* (after tne Reporting, in parallel with Re-work and 
l~t~ CW _ P ■ _ If f rrors are discovered in the product after the inspection 
_ _ y 3 " s — um P ~ l --- , out prior to ~he product oemg delivered to the 
yp ir ths process, then an inspection process step is in place 

:,J '--uture the information on these errors the same as if they h!d*be«n 
;’; S r OVe r ed durinQ the inspection meeting. Errors found after' the 
;: S r?! tiQn y etin 9 are either corrected prior to delivery of the product 
7'_ - _yy e; — s.tep in the process, or else formal Probletn Reports aoainst 
_ ~ _ ~ 2 ~ r ~ en - er2i ^ in the Configuration Management system. 

yyyy teria -after Re-mspection) . Inspection process must satisfy 
°f measurable actions before development can proceed to the ns^ 
;y: y. th s software engineering process. These actions shall insure 

r a ;. aii major errors have been corrected, or documented via formal 
r h od i erne Reports. 

! ^ * i «_tdm needs to be added under "Describe the benefits of t K »= 
ru-rmai inspection process". The item is: 

_y 1 y near immediate feedback to the process developinn the product 
’-inOar^ inspect! ohs of escapes that are occurring. This information 
— ^ ~ t ~ r- - proc-tx improvement for the prior proces—-. 
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f i nr ; 5. 1 i t-E: m s n“?d 0 d to b e added undsr "Describe the 0 - - 
planning and preparation in che inspection process— Plan-Tir-g" . 

Thp additional items a r e : 

— Determine the number* of parts of a product tc inspect an a. = : ■ ; 1 c 

i n = r ecti cn meed no so as tc* not exceed the 2 hour meet ire time dm 

— Determine all inspectors whose support is esse^dal + g *~ “he success 
ct the meet i no and identify those inspectors as mancatorv f c r- t r e 

l nepect i jp meet i no . 

— I - sp*ec t i c ,n meetings s^all be scheduled tar enough in the dr__‘~s tc 
allow at least the minimum lead time defined bv the in specs ion 


Additional item: needed to be added under "Describe the role d 
planning and preparation in the inspection process — Prepar at i on " , 

The additional item is: 

— All errors shall be documented -For presentation during the meeting. 

Additional items needed to be added under "List the review roles assumed 
during the inspection". Also, one role is not required. 

The role not required: 

— Recorder (Errors are documented prior to inspection meeting by 
inspectors, collected by moderator. Moderator responsible for 
producing final report, including summary of actions.). 

Additional roles needed: 


— Representative from a previous step in the s/w engineering process 
(i.e.c requirements analysts for design/code inspection). 

— Representative form the next step in the s/w engineering process 
vi.e, , integration or independent test for design/code inspection* . 

Information* presented under "List the types of checklists 
; K at ca^ be used before, durinc the inspection and to r follow-up" 
coulc be presented in a number of ways. Checklists should promote 
t following general types of errors: 

— Requirements compliance 

— Interfaces 

— Logic 

— Standards compliance 

— Performance 

— Readability 

Disagree with the fallowing characteristic? under "Character i st; cs 
o-f an inspection meeting": 

— recorder records defects on Inspection Defect Log 

(would simply say that moderator is responsible for defects to be 
recorded on Inspection Defect Log) . 

Additional item needed to be added under "Basic rules for Code 
Inspections". The additional item is: 

— Team of inspectors owns the product after the inspection and 

are jointly responsible for all errors not detected. Quality owner 
transfers from producer to inspection team at the end of the inspec 
meeui ng . 

. Disagree ’with item under "Module 2 Objectives, 

Item: 3 Compare the differences between inspection and walkthrough 
procedures, - Software Inspections, - Disadvantages". 

— Ar- inspection does not have to be ke vy ed to developer s viewpoint. 


i ■+ iji 
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Proper inspections will have other organization roles represented 
other than the developer. 


11. Additional items needed to be added under Module 3 Objectives, 

Item 1 . A Moderator role description. The additional items are: 

— Do cumene and classify errors 

— Disposition errors as to action to be taken 
— Assign errors di sposi ti oned tor correction to the author 

verity j personally or by delegation, that all sr r ors disposition 
tor correction are actually corrected prior to .authorizing delivery 
cc one" nex t step in the software engineering p^ocsss. 

12. Disagree with role of Recorder under Module 3, Item l.C. These actions 
are performed oy a combination of inspectors (records errors found in 
preparation) and moderator (records errors found real time in inspection 
meeting* collects other recorded errors, and prepares summary meeting 
rsnorts) . 


13. i * ■ Module 3, Item 3, impacted by previous comments, especially expanded 
moderator role, additional roles, and deletion of recorder role. 

- 4 - In Module 3, Item 4, the data required is insufficient. Refer to Section 
3.~ Required Data of the Software Inspection Process Standard produced 
via SQA Standards RTOP under this contract (Southwest Research Institute 
should have been given a copy of this document previously). 

^5. In Module 3, Item 6, impacted by previous comments, especially expanded 
moderator role, additional roles, and deletion of recorder role. 

- - ■ 1 r - Module snauld also acknowledge the importance of inspecting software 

~ - an organizational approach to ensuring process stabi 1 i ty /improvement . 


negie Mellon Code Inspection Model For IBM/FSD/Houstor 


1. cose s steps adequate. Exit criteria should be last. Missing items 
±i e r S"W o^k, post meeting errors, collection of inspection meeting 
reports, submission of summary data to database, extraction of reports 
-:^cmt: database , summary metric data, FAC I /Cl summary data, etc. 

'■ Defined Roles": 

Multiple managers involved. Development manager assigns 
individual to work a particular change instrument (Change 
Request, CR, or Discrepancy Report (DR)). Verification and 
Requirements Analysis managers maintain lists that relate 
responsible individual to specific requirements. Librarian 
contacts assigned individual when inspection package is 
received. Manager involved only in special situations 
(excessive workload, illness, etc.) 

£11 managers of individuals parti ci pati ng in the inspection 
receive reports on Major Errors discovered. 

Development manager signs off as part of promotion (i.e., 
formal submission to can-figuration controlled library 
process via Configuration Management Data Base controlled 
bui i d process) . 

-^oducer (author): Disagree with "does unit level testing prior to 

inspection". .Unit testing (currently called development 




testing) is done after the inspection package is prepared , 
and in parallel with setting up the inspection, di st^i hut: c“ 
□f materials, and preparation for the inspection meeting 
by the moderator and other inspectors. Depending on t K e 
timing for a specific change/inspection, unit tests may 
have been executed. However, the ideal situation would 
be for the inspection meeting to occur prior to executin '" 1 
unit tests. 

" Moderator? T ^e single most important role. Primary role is to 

as dr ess unique issues, control the meeting, insure results 
are recorded , insure correction actions are assigned, 
insure correction actions are complete, and that the code 
inspected is ready for promotion via build process. 
Responsible to insure that the source promoted to the 
conf iguration controlled library via build process is 
identical to that inspected (if no correction) , or the 
version reviewed to insure all corrections were dc^e 
correctl y . 

NOTE: Moderator does not have to be from the organization 

of the Producer. Moderator Pool includes Producers, 
Requirements Analysts, and Independent Verifiers. 
Moderators must have special education. 

— Librarians Performs para technical tasks including: 

— arranges the inspection meeting time and place, 
including making sure inspectors and moderator 
can support 

elevates any scheduling issues to moderator 
-- schedules inspection room and distributes materials 

— post inspection meeting (after build inputs have been 
submitted), files meeting records. 

— enters key information in database 

produces regular and special request reports from database 

recorder: Role does not exist. Individual inspectors prepare error 

descriptions prior to meeting. Moderator responsible for 
recording issues discovered real time in meeting, plus all 
summer y reports . 

designer /Testers Confusion of these ^ o I e s . 

is the Designer, so this is not a 
doing unit testing would normally 
Peer . 

Independent Tester: The independent detail verifier brings a 

" non— devel opment /out si der " view and objectivity to the 
inspection process. Acts as a reviewer (May also have 
another role, e.g. , reader) . 

Consumer: For design/code inspection, this role is occasionally 

performed by members the Test and Operations team. 

However, normally, thii function is performed by the 
Requi rements Anal yst . 

Per design/code inspections, no manager or non— IBM 
contract personnel are allowed to participate in the 
inspection process. Mote that excludes the NASA 
customer (”DSD) . Included are IBM'ers and IBM subcon tr act or 


Generally, the Producer- 
unique role. Tester 
be addressed as a 
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personnel. Inspection process for desiqn/ccde p^od^ceo 
by sub contractor personnel or IBM personnel a.^e Sub ’eoted 
to identical process. 

~ o~ Detail V eritic 3 1 i o ^ T est Insoecti one , the c on s _.n o*- - 
^ole is tilled by the NASA customer (FDSD personnel.- 
and other non- 1 participants like NASA (or non-iB>t 
contractors) requirements owners (called principle 
function owners). 
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APPENDIX G 


VALIDATION PLAN 



NASA CODE INSPECTION INSTRUCTIONAL VALIDATION 


VALIDATION PLAN 


1. Content Validity of Materials (Content) 

A. Compare Objectives (NASA and Carnegie Mellon) 

Purpose: The purpose of this activity is to determine die objectives taught by the Carnegie 
Mellon Code Inspection Course and compare these with current NASA objectives to 
find similarities and differences in terms of content, specifically what objectives are 
missing or extraneous in the Carnegie Mellon Course. 

Input: Objectives (4 sources) 

1. Objective oudine provided by Carnegie Mellon 

2. Objective description (answers) provided by SwRI 

3. Objective description provided by IBM analysts 

4. Description of current code inspection process (NASA Software Inspection 
Process Standard, Software Formal Inspections Guidebook) 

Activity: Compare NASA and Carnegie Mellon objectives 

Output: Data for final report 

Test instruments 

B. Compare Models for Code Inspection (NASA and Carnegie Mellon) 

Purpose: The purpose of this activity is to compare the model for code inspection used by the 

Carnegie Mellon Code Inspection Course and the model used by NASA for code in- 
spection to find similarities and differences, specifically to determine if the two 
models are similar enough to make the Carnegie Mellon Course content relevant for 
use with NASA analysts. 

Input: Carnegie Mellon and NASA objectives for code inspection as provided by Carnegie 

Mellon 

NASA model for code inspection as provided in the following documents: NASA 
Software Inspection Process Standard, Software Formal Inspections Guidebook 
Activity: Compare NASA and Carnegie Mellon models for code inspection 

Output: Data for final report 

Test instruments 


2. Effectiveness of Materials (Presentation Strategy) 

(3 NASA analysts use the Carnegie Mellon Code Inspection Course) 

A. Knowledge of Information (Instructional Modules) 

- pretest/posttest (paper/pencil test over Carnegie Mellon objectives) 

Purpose: The specific purpose of this activity is to measure the effectiveness of the instruction- 
al modules and look in general at the effectiveness of this type of course for teaching 
the content, code inspection, as well as other similar processes. Analysts will be 
administered a pretest prior to interacting with foe instructional modules (Carnegie 
Mellon Course) to determine their prior level of knowledge of code inspection. A 



posttest will be administered after interacting with the instructional modules to 
determine how much was learned from die modules. Gains in scores from the pretest 
to the posttest will be examined. 

Input: Pretest 

Posttest 

Demographic data 

5 analysts (pretest) (not highly experienced in code inspection) 

3 analysts (posttest) 

Activity: Select 3 analysts to participate in validation (pretest) 

3 analysts participate in the instructional module portion of the course 
Test analysts after instructional modules (posttest) 

Output: Selection of 3 analysts 

Record of prior knowledge of code inspection 
Gain in score attributed to instructional modules 

Note: The pretest will determine from the original 5 analysts which 3 will participate in the 

validation process. 

B. Application of Information (Simulated Code Inspection) 

- posttest (paper/pencil test over Carnegie Mellon objectives) 
course feedback after practice code inspection 

Purpose: The purpose of this activity is to measure die effectiveness of die simulation 
(simulated code inspection) as a teaching tool. After interacting with the simulation 
portion of the course, analysts will be given die same posttest (as used in Part A) to 
determine if there is any gain in their scores after participating in the simulation. In 
addition, comments will be noted regarding die course assessment of the analysts 
performance in the simulation. 

Input: Posttest 

3 analysts 

Activity: 3 analysts participate in the simulation portion of die course 

Test analysts after simulated code inspection (posttest) 

Output: Gain score attributed to simulation 

Program comments regarding performance 

C. Overall Course 

- analyst self-report (questionnaire) (interview) 
evaluator report (observations) (opinions) 

Purpose: Upon conclusion of the Carnegie Mellon Code Inspection Course (instructional 
modules & simulation), analysts will be given a questionnaire asking for their subjec- 
tive response to the course. Following are the purposes of the questionnaire: 1) to 
determine whether analysts liked this method of instruction, 2) to determine if 
analysts felt they learned from the course, 3) to identify areas where difficulties 
occurred, 4) to find out what they thought the strengths and weaknesses of the course 
were, 5) to determine opinions on whether weaknesses could be overcome, as well 
as possible suggestions to overcome them. The interview will be an extension of this 
line of questioning to give the analysts an opportunity to further express their 
opinions about the course and its potential for use by NASA analysts. 

The evaluator's report will document observations of how the analyst performed 
during die course. Observations will include items such as problems encountered, 



analyst comments, and time spent on various parts of the instruction. Subjective 
opinions of the observer may be included for the purpose of documenting or 
clarifying events occurring during the validation process. 

Input: Analyst questionnaire 

Analyst interview 
Evaluator observations 
Evaluator opinions 
Activity: A dminis ter questionnaire 

Interview analyst 

Observe analyst using the code inspection course 
Output: Analysts’ objective responses to the course 

Observer’s documentation of what occurred during the course 


NOTE: Part 2 will take place over two days using analysts provided by NASA: 

Day 1: Set up die computer system 

Use materials with 1 analyst (approximately 3-4 hours) 

Day 2: Use materials with 2 analysts (approximately 3-4 hours each) 

NOTE: Instruments to be developed: 

1. Pretest/Posttest 

Purpose: The purpose of this test is to measure the effectiveness of the instructional modules 
and the code inspection simulation. The pretest and posttest will be the same instru- 
ment. This test will be comprised of content taken direcdy from the Carnegie Mellon 
objectives addressed in the five instructional modules of the course. Test answers 
will be reverse engineered from the Carnegie Mellon course by SwRI. 

2. Analyst Questionnaire 

Purpose: The overall purpose of the analyst questionnaire is to determine the potential for 
using this course and/or the feasibility of this type of instruction for code inspection 
or other similar processes (content) relevant to NASA needs. The questionnaire will 
consist of ite m s for the analyst to respond to regarding their opinions about the 
Carnegie Mellon Course. 

3. Interview Questions 

Purpose: The purpose of the interview is to give each analyst further opportunity to express 
his/her opinions regarding strengths and weaknesses, and the potential for this course 
or this type of instruction for teaching code inspection and other content relevant to 
NASA. 

4. Observation Form 

Purpose: The purpose of the observation form is to provide additional descriptive data which 
may aid in explaining results found in the data collected. The observation form will 
be used by the observer to document any problems encountered, any comments made 
by die analyst either positive or negative, and how long the analyst spent during parts 
of the course. 



Note: The observer will play an impartial role in documenting events, however, a 

secondary role of the observer is to facilitate analysts in staying on task to insure 
validity of comparisons between the three analysts involved. 

5. Demographic Data Sheet 

Purpose: The purpose of the demographic data sheet is to provide information which might 
help analyze results found in the data collected. The data sheet will be used to 
provide background information about the analysts participating in the validation 
study. 


3. Final Validation Report 

A. Purpose of the Study 

The purpose of this study is to validate the instructional effectiveness of the Carnegie Mellon 
Code Inspection Course. This code inspection course validation provides a case study for 
exploring process simulation training with a subject matter domain of software. 

B. Materials 

The materials used in this study will be listed and briefly described including the Carnegie Mellon 
Course itself, as well as the pretest/posttest, analyst questionnaire, interview questions, 
observation form, and the demographic data sheet. 

C. Procedures 

The procedures used in collecting the data described in Part E (Results) will be stated. 

D. Subjects 

The subjects (analysts) participating in this study will be briefly described. This description will 
give general information about die subjects, based on die information they provide on their 
demographic data sheets, as well as any observations made by observers during the validation 
process. 

E. Results 

Results will be reported for two areas including content and instructional effectiveness. Results 
will be presented based on the data gathered by the instruments described previously. 

1) Content Data 

a) Objectives 

b) Models for Code Inspection 

2) Instructional Effectiveness Data 

a) Pretest/Posttest (Knowledge of Information) 

b) Posttest (Application of Information) 

c) Course feedback after practice code inspection (Application of Information) 

d) Analyst Self-Report (questionnaire, interview) 

e) Evaluator Report (observations, opinions) 

F. Conclusions and Recommendations 

SwRI will synthesize and interpret the results and summarize the subjective conclusions. SwRI 
will provide a professional recommendation based on these results and conclusions. Recommen- 



dations will be made as to the use of this specific course, as well as this type of instruction in 
general. 

G. Limitations 

Any factors seen as limitations on this study which may affect the results will be clearly stated 
so that knowledge of this information can be used in interpreting the results. One example of 
such a limitation is that the scope of this project does not allow for a true empirical study to be 
implemented. This is not possible with the limitation of only three analysts using the course. 
Due to the small number of subjects, statistical manipulation of the data is not appropriate, 
therefore data gathered will be descriptive in nature. 

H. Appendix 

The appendix will contain copies of all pertinent documents to this study. Such documents will 
include instruments, Carnegie Mellon and NASA objectives, Carnegie Mellon and NASA models 
for code inspection, and any other documents relevant to foe validation process. 




APPENDIX H 


OUTLINE OF "A CURE FOR THE COMMON CODE" 




NASA COPE INSPECTION INSTRUCTION AL VALIDATION 
STRUCTURE OF THE CODE INSPECTION COURSE SOFTWARE: 

Auditorium 

- motivational presentation (why software quality is important and how inspections improve the 
quality of software) 

information on the company Ultimex and course the user is about to take 
Training Room 

learn how to navigate through the simulated world 

- access to instruction on inspections and group processes 
Instructional Modules (5) 

1. Formal Inspections: Purpose and Process 
The Purpose (4 minutes) 

The Process (9 minutes) 

The Process (3 minutes) 

Conducting An Inspection (3 minutes) 

Guidelines For Success (3 minutes) 

2. Inspection Types and Differences 
Inspections 

Definition 

Function 

Walkthroughs 

Definition 

Function 

Reviews 

Definition 

Function 

Advantages/Disadvantages 
Inspections 
Walkthroughs 
Reviews/Audits 
Quality Assurance Techniques 

3. Inspection Roles and Pitfalls 
Moderator 

Role (14 minutes) 

Problems (up to 10 minutes) 

Attack On Producer (2 minutes) 

Moderator Dominates (2 minutes) 

Producer Problem Solving (2 minutes ) 

Follow-Up Communications (2 minutes ) 

Pitfalls (2 minutes) 

Helpful Hints (up to 20 minutes) 

What if: (20 items) 

Checklists (up to 10 minutes) 

Before 

During 

After 

Producer 

Role (5 minutes) 



Problems (up to 6 minutes) 

Attack On Producer (2 minutes) 

Producer Problem Solving (2 minutes) 

Pitfalls (2 minutes) 

Reader 

Role (3 minutes) 

Problems (up to 4 minutes) 

Reader Too Fast (2 minutes) 

Pitfalls (2 minutes) 

Recorder 

Role (7 minutes) 

Problems (up to S minutes) 

Recorder Too Slow (3 minutes) 

Pitfalls (2 minutes) 

4. Inspection Tools and Forms (practice code inspection) 

How To Get Help (2 minutes) 

Office Window Environment (3 minutes) 

Tools For Code Analysis (up to 9 minutes) 

Hypertext Tools Demo (4 minutes) 

Code Debugger Tools Demo (5 minutes) 

Electronic Mail (not available) 

Practice With The Tools 

5. Inspection Communications 
Group Process (up to 23 minutes) 

What Is A Group? (5 minutes) 

Different Types of Groups (5 minutes) 

Group Communication (5 minutes) 

Special Problems (8 minutes) 

Conversational Interface (6 minutes) 

Library 

- contains text, graphics, video, and audio materials 
Videotapes (11 available) 

Card Catalog 

- Forms 

- Style Manuals 

- Articles 

- Slides (not available) 

User's Office 

- examine code that will be the subject of a later inspection 

- purpose is to teach the importance of preparation in an inspection and to give the student 
experience in preparing for an inspection 

- tools available to help prepare for die inspection (hypertext system, source level debugger) 

- secretary asks user to choose role for the inspection (moderator, reader, recorder) 

Conference Room 

- location for the simulated code inspection 

- code, specifications and error report forms complete with hypertext links are available here during 
the simulation of the inspection 



Secretary 

user does not have access to this room 
Coffee Room 

upon conclusion of the inspection, the user is given performance feedback here 





Copies of this publication have been deposited with the Texas State Library in 
compliance with the State Depository Law. 


